Eduzan

Learning Hub

Eduzan

Eduzan / 01 Quantitative Methods

QM 7 – Big Data Projects

Worked examples are fully visible. Check-yourself items are study aids you can reveal one at a time.

Investment managers keep searching for signals that competitors have not yet found. Structured inputs, meaning the numeric fields that already sit in tidy rows and columns, are available to everyone and are therefore a thin source of edge. What is not equally available is the enormous stream of written and recorded material that surrounds every listed company: filings, transcripts, newswires, forum posts, clickstreams. A manager who can turn that material into a usable number before anyone else has an advantage.

That is the whole point of this reading. It is not about machine learning algorithms as such. It is about the pipeline that turns raw, messy, mostly unstructured material into a matrix an algorithm can read, and about how to tell whether the resulting model is any good. A typical application is scoring the tone of a 10-K filing and feeding that score, alongside conventional accounting ratios, into a larger forecasting model.

What makes data big

Big data is distinguished from traditional data by three properties, conventionally labelled the 3Vs, to which a fourth is added whenever the data are used for inference or prediction.

Volume. The sheer quantity. The reference point used in the curriculum is the US Library of Congress, whose archive of digital and physical artefacts runs to hundreds of terabytes, where one terabyte equals 1,024 gigabytes, which in turn equal 1,048,576 megabytes. The average company in most US industry sectors now collects more than that. The working units in big data conversations have moved on to petabytes and exabytes, where one exabyte equals 1,024 petabytes, which equal 1,048,576 terabytes. The grains of sand analogy makes the jump concrete: if a megabyte is a tablespoon of sand, a petabyte is a beach 1.6 kilometres long and an exabyte is a beach stretching roughly 1,600 kilometres.
Variety. The range of sources and formats. A firm now handles structured, semi-structured and unstructured material drawn from inside and outside its own walls: transaction records, user-generated text, images and video, social media, sensor readings, web and mobile clickstreams, and spatial-temporal data. Exploiting that range raises legal and ethical questions, data privacy above all.
Velocity. The rate at which new data arrive. Large organisations can collect several petabytes an hour. On the unstructured side, more than one billion new tweets, each a message of 280 characters or fewer, appear every three days, and five billion search queries are run daily. Data arriving this fast are described as data-in-motion, and the patterns inside them are moving targets in a way that patterns in data-at-rest are not.
Veracity. The credibility and reliability of a source. This fourth V matters most precisely because big data is drawn from so many places. Social platforms, blogs and forums are saturated with spam: by some estimates 10% to 15% of that content is entirely fake, and more than 20% of everything published on the worldwide web is estimated to be web spam. Clickstream records are similarly noisy. Extracting deep semantic meaning from text also remains hard in places, despite real progress in natural language processing.

Veracity is the V that separates a research exercise from a production investment process. Volume, variety and velocity describe the supply of data. Veracity asks whether the supply deserves to be trusted, and big data magnifies the old problem of telling quality apart from quantity.

Why this matters for fintech

The practical consequences show up across investment management. Creditworthiness assessment, historically driven by structured financial ratios, is being enriched with text pulled from financial statements, news coverage and call transcripts. Client segmentation now draws on stated views and preferences expressed publicly, to the extent that privacy agreements permit, and not only on transaction history. Fraud detection and risk management both benefit from the wider net.

Textual big data is valuable because it carries two things that numbers do not carry directly. Topics are what people are discussing: a firm, an industry, a specific event. Sentiment is how they feel about it, expressed as positive, negative or neutral. One United States study found that positive sentiment on Twitter predicted the direction of the Dow Jones Industrial Average up to three days ahead with nearly 87% accuracy.

Why a stated accuracy figure like 87% should not be taken at face value

An accuracy number means nothing until you know the class balance of the sample it was measured on. If an index rises on 60% of days, a model that always predicts up scores 60% without containing any information at all. Accuracy also says nothing about which errors were made. A model that never misses a rally but also never flags a drawdown can post a high score while being useless for risk control. Later in this lesson the F1 score is introduced precisely because accuracy alone is unreliable whenever the two classes are unequal in size, and the confusion matrix is introduced because the composition of the errors matters as much as their count.

Two pipelines are needed, because the first four steps differ depending on whether the raw input is already structured. Both pipelines converge: whatever the starting material, an algorithm is eventually trained on a rectangular matrix of numbers. The text pipeline simply has more work to do before it reaches that point.

The traditional pipeline, starting from structured data

Conceptualisation of the modelling task. Decide what the model is supposed to output, for example whether a share price will be higher or lower one week from now, who will consume that output, and how it will sit inside an existing or new business process. Getting this wrong makes everything downstream irrelevant.
Data collection. Gather the mostly numeric inputs from internal and external sources. These usually arrive already tabular, with columns of features, rows of instances, and a value in each cell.
Data preparation and wrangling. Cleanse the raw material, then preprocess it. Cleansing deals with things such as missing entries and values that fall outside a sensible range. Preprocessing covers extracting, aggregating, filtering and selecting the columns that will actually be used.
Data exploration. Exploratory data analysis, feature selection and feature engineering.
Model training. Choose a method or methods, measure how the trained model performs, and tune it accordingly.

These steps are not a one-way conveyor belt. Model building is iterative: what is learned on one pass feeds the next, and the next pass may begin all the way back at reconceptualisation.

The text pipeline, starting from unstructured data

News articles, social posts, internal and external documents such as published financial statements, and open data sources arrive with no structure at all. The first four steps are replaced, and their purpose is to manufacture structured output from unstructured input.

Text problem formulation. Decide how the classification problem is to be framed, and name the exact inputs and outputs. A common framing is to compute sentiment scores, which are structured, from text, which is not. It must also be settled in advance how the classification output will be used.
Data (text) curation. Obtain the relevant external text through web services or through web spidering programs, meaning scraping or crawling routines that pull raw content from a source, usually web pages. Where supervised learning is intended, the text must also be annotated with dependable labels for the target variable. Experts might have to mark each analyst assessment of a stock as bearish or bullish before anything can be trained on it.
Text preparation and wrangling. The cleansing and preprocessing that converts a stream of unstructured characters into a form that conventional, structured-input methods can consume.
Text exploration. Visualisation, for instance through word clouds, together with feature selection and feature engineering on the text.

The output of the text pipeline, a sentiment prediction score for example, can then be used directly for forecasting or combined with structured variables inside a larger model. The reason the first four steps diverge is that they must cope with the characteristics of big data, and in a text project it is variety and veracity that bite hardest.

Figure 1: Two pipelines, one destination

The two routes differ in their first four steps only. Both end in the same training stage, because both must arrive at a numeric matrix before an algorithm can be fitted.

Example 1 · Worked

LendALot Corporation is a business-to-consumer lender that has always paid an outside firm to score the creditworthiness of applicants. Machine learning now allows assessment to go beyond the traditional repayment history and ability to repay measures built from structured fields, so LendALot wants an in-house capability. The plan is phased: begin with structured sources, then add unstructured text later. Paul Wang leads the new analytics team.

1. Name one decision Wang must take at each of the five structured-data steps.

Solution. Take them in order.
Conceptualisation. How the model output is to be specified, for instance as a binary creditworthy or not creditworthy label, who will use it, and where it sits in LendALot business processes.
Data collection. Whether credit scoring will run on internal fields, external fields, or a combination of the two.
Data preparation and wrangling. What cleansing and preprocessing is required. Cleansing will have to settle missing entries and extreme values; preprocessing will involve extracting, aggregating, filtering and selecting the columns that matter.
Data exploration. Which exploratory methods suit the data, which features to carry into the model, and which features have to be engineered from existing ones.
Model training. Which algorithm to run. Because labelled outcomes exist for past loans, the choice sits inside the supervised family. Wang must also decide how fit is measured and how the model is validated and tuned.

2. Later, Wang tells the team to improve the scoring model using the free-response fields on the loan application. Which process step is that instruction?

Solution. Text problem formulation, the opening step of the text pipeline. Wang has specified the input, which is free-response text, and the intended contribution to the output, which is a better creditworthiness score. He has not yet gathered any text, so curation has not begun.

3. State two things the team must confirm at the text curation step.

Solution. The text is internal, coming from LendALot own loan applications, so curation is about the integrity of that internal capture rather than about scraping. First, the team must confirm that the free-text comment fields on the application form were correctly implemented and switched on. Second, if completing those fields is optional, the team must confirm that enough applicants actually filled them in for the responses to support analysis.

4. State two things the team must do at text preparation and wrangling.

Solution. Text cleansing and text preprocessing. Together these two tasks are what convert an unstructured stream into structured values that conventional modelling methods can accept, and neither can be skipped.

Preparation and wrangling is where most of the elapsed time in a big data project goes, and the quality of what comes out of it caps the quality of everything that follows. It is also the stage where domain knowledge is least optional: the specialists who understand what the fields actually mean have to be in the room.

The stage sits between collection and exploration and splits into two tasks. Cleansing examines the raw material and puts right the errors in it, because raw data are rarely complete enough or clean enough to train on. Manually keyed records carry omissions, duplicates and plain mistakes; automatically recorded data carry the same defects for different reasons, such as server failures and software faults. Preprocessing then transforms the cleansed material into the shape a model can consume, dealing with outliers, deriving new variables from existing ones and rescaling.

Figure 2: Where preparation and wrangling sits

Cleansing removes errors, preprocessing reshapes what survives. Everything downstream inherits whatever is left uncorrected here.

Before collection begins

Nothing should be downloaded until the problem has been stated, the objectives defined, the useful data points identified and the model conceptualised. Conceptualisation works like a blueprint on a drawing board: a plan that can be revised, but a plan that has to exist. It fixes whether the model will be supervised or unsupervised and settles what will be collected from where.

Collection itself means finding and downloading raw material from one or several places. Databases are the most common primary source, so queries should be built with a database administrator. Schemas embed assumptions and exceptions, and the safe course is to have the architecture explained before pulling anything. Data also arrive as spreadsheets, comma-separated values files and plain text files. Where documentation exists, typically a Readme file supplied alongside the raw data describing what the file contains, it must be read, because it is what allows the fields to be interpreted correctly.

Third-party vendors are the alternative. External material is usually reached through an application programming interface, a defined set of communication methods between software components, or delivered as files. Buying saves the time and cost of preparing data yourself, but vendor contracts carry a price, so the decision turns on the trade-off between time, money and accuracy given the project constraints.

Two further considerations decide the internal against external question:

Fit to purpose. Projects built on internal user behaviour cannot use external substitutes. Understanding traffic on a company website requires that company own record of visits and click frequency. External data earn their place where the requirement is generic, such as the demographics of a region or traffic on a public service.
Loss of edge. Vendor cleansing can mask or destroy exactly the underlying pattern a particular end use depends on. That is often where alpha lives, so buying a dataset off the shelf can cost you the informational advantage you were trying to buy. What remains is that every user merges, combines and models the same file differently, so value can still be extracted in different ways.

The six error types

Cleansing structured data means hunting for six recognisable defects. The illustration used in the curriculum is a customer table for a United States bank, with one row per customer and one column per feature.

Raw customer records before cleansing

Row	ID	Customer	Gender	Home state	Holds a card	Birth date	Salary	Income from elsewhere
2	1	Mr. ABC	M	VA	Y	12/5/1970	$50,200	$5,000
3	2	Ms. XYZ	M	NY	Y	15 Jan, 1975	$60,500	$0
4	3	EFG		CA	N	1/13/1979	$65,000	$1,000
5	4	Ms. MNO	F	FL	Do not know	1/1/1900
6	5	Ms. XYZ	F		Y	15/1/1975	$60,500	$0
7	6	Mr. GHI	M	TX	N	9/10/1942	NA	$55,000
8	7	Mr. TUV	M	CT	Y	2/27/1956	$300,000	$50,000
9	8	Ms. DEF	F	British Columbia	N	4/4/1980	$55,000	$0

Row numbers follow the source spreadsheet, in which row 1 holds the column headings.

Incompleteness. A required value is simply absent. The remedy is to look for the value in another source. Where it cannot be recovered, missing entries and NA markers, meaning not applicable or not available, are either dropped or flagged as NA so that they can be deleted or replaced by an imputed value during exploration. The usual imputations are the mean, the median or the mode of the variable, or a simple assumption of zero. Rows 4, 5, 6 and 7 are all incomplete, missing a value in gender, salary, other income, the salutation part of the name, or state.
Invalidity. A value lies outside any meaningful range. Administrative records are the check. Row 5 records a date of birth of 1/1/1900, outside a plausible human life span.
Inaccuracy. A value is present and within range but does not describe reality. Business records and administrators resolve it. Row 5 again offers the example: the credit card field reads Do not know, whereas every person either holds a card or does not.
Inconsistency. Two fields contradict one another, or contradict the world. A second source has to arbitrate. Row 3 carries a female salutation alongside a gender code of M.
Non-uniformity. The same information is recorded in different formats. Conversion to one agreed standard is the fix. The date of birth column runs through several layouts, and the salary column is ambiguous too, because a dollar symbol could denote United States dollars, Canadian dollars or others.
Duplication. The same observation appears more than once, and the extra copies are removed. Row 6 repeats the name and date of birth of row 3, so it refers to the same customer.

Cleansing is expensive. It combines automated rule-based and pattern recognition tooling with human inspection, working through the table row by row and column by column for each of the six defects. Analysis software such as SPSS helps by exposing metadata, meaning data that describes the properties of other data, which gives investigators a starting point. How much cleansing is justified depends on the business value of the project. Where errors cannot be resolved with the resources available, the affected rows can be dropped, and whether that is acceptable depends on scale: removing roughly 100 rows from a dataset of more than 10,000 will not matter much, whereas in a dataset under 1,000 rows every row may count and heavy deletion becomes damaging.

The same records after cleansing

Row	ID	Customer	Gender	Home state	Holds a card	Birth date	Salary	Income from elsewhere
2	1	Mr. ABC	M	VA	Y	12/5/1970	USD 50200	USD 5000
3	2	Ms. XYZ	F	NY	Y	1/15/1975	USD 60500	USD 0
4	3	Mr. EFG	M	CA	N	1/13/1979	USD 65000	USD 1000
5	6	Mr. GHI	M	TX	N	9/10/1942	USD 0	USD 55000
6	7	Mr. TUV	M	CT	Y	2/27/1956	USD 300000	USD 50000
7	8	Ms. DEF	F	British Columbia	N	4/4/1980	CAD 55000	CAD 0

ID 4 has gone entirely, the duplicate of ID 2 has gone, dates are uniform, currencies are explicit, and the missing salary for ID 6 has been resolved to USD 0.

Five preprocessing transformations

Extraction. Derive a new variable from an existing one so that it can be analysed and trained on. Date of birth is not directly usable, so an Age variable is extracted by taking the number of years between the reference date and the date of birth.
Aggregation. Combine two or more related variables into one. Salary and Other income are summed into a single Total income field.
Filtration. Identify and drop rows the project does not need. Row 7 holds a non-United States state, and the dataset covers customers of a United States bank who are required to have a United States address.
Selection. Drop columns that are intuitively unnecessary. Name and Date of birth are not required for training, since ID identifies each observation and the newly extracted Age replaces date of birth. This is not the same thing as feature selection, which comes later and is statistical rather than intuitive.
Conversion. Force each variable into the correct type: nominal, ordinal, continuous or categorical. Prefixes and suffixes such as currency symbols are stripped first. In this table both Name and State are nominal variables, Salary and Income are continuous, and Gender and Credit card are categorical with two classes apiece. Had row 7 been retained rather than filtered out, its salary would first have had to be converted into United States dollars. Conversion also covers adjustments for the time value of money and for time zones where those apply.

Outliers

Outliers must be found before anything is done about them, and domain knowledge decides what is done. Each candidate is examined and then either removed or replaced by a value imputed statistically. In the cleansed table, the salary of ID 7 sits far above the upper quartile and the salary of ID 6 sits far below the lower quartile, so both are outliers on that variable. Note what happens once aggregation is performed: after Total income is created, ID 6 is no longer an outlier at all, because the income was there all along and merely sat in a different column.

Two detection rules are standard. Where the data are normally distributed, a value more than 3 standard deviations from the mean is generally treated as an outlier. Where the distribution is unknown or non-normal, the interquartile range is used instead. The IQR is the gap between the 75th and the 25th percentile values. The bounds are set at 1.5 times the IQR beyond the quartiles, so the upper bound is the third quartile plus 1.5 times IQR and the lower bound lies 1.5 times IQR below the lower quartile. Using a multiple of 3.0 in place of 1.5 identifies extreme values rather than merely unusual ones.

Note on the source: the curriculum writes the lower bound as −1.5 × IQR + 2nd Quartile. The 2nd quartile is the median, so read literally that expression would place the lower fence below the median rather than below the 25th percentile. The intended and conventional rule, and the one consistent with the worked identification of ID 6 as a low outlier, measures the lower bound from the lower quartile.

Two handling methods follow. Trimming, also called truncation, simply deletes extreme values: a 5% trimmed dataset has had the highest 5% and the lowest 5% of values removed. Winsorisation keeps every observation but replaces each extreme value with the nearest value that is not an outlier, so large outliers take the maximum non-outlier value and small ones take the minimum.

Scaling

Scaling shifts and rescales a feature so that its range becomes comparable with the ranges of other features. Age and income live on wildly different scales, and a training set built from such variables is heterogeneous in a way that hurts methods such as support vector machines and artificial neural networks. Those methods work better when every variable occupies the same range. Outliers must be removed before scaling, not after.

Normalisation rescales the variable onto the interval from 0 to 1.

Standardisation centres the variable on a mean of 0 and rescales it to a standard deviation of 1.

The choice between them is governed by two properties. Normalisation is highly sensitive to outliers, because the minimum and maximum define the whole scale, so outlier treatment must come first; on the other hand it can be used when the shape of the distribution is unknown. Standardisation is comparatively insensitive to outliers, since it depends on the mean and standard deviation rather than on the extremes, but it presumes the data are normally distributed.

Example 2 · Worked

Apply the preprocessing transformations to the cleansed customer table above, then scale the resulting income variable.

1. Build the transformed table.

Solution. Extract Age from date of birth, aggregate Salary and Other income into Total income, filter out row 7 because British Columbia is not a United States state, and select away Name and Date of birth. Working the aggregation:
ID 1: 50,200 + 5,000 = 55,200.
ID 2: 60,500 + 0 = 60,500.
ID 3: 65,000 + 1,000 = 66,000.
ID 6: 0 + 55,000 = 55,000.
ID 7 is dropped at this point as a salary outlier, and ID 8 has already been filtered out. The result is:

After transformation

Row	ID	Age	Gender	Home state	Holds a card	Total income
2	1	48	M	VA	Y	55200
3	2	43	F	NY	Y	60500
4	3	39	M	CA	N	66000
5	6	76	M	TX	N	55000

The ages are the whole years between each date of birth and the project reference date.

2. Normalise Total income across these four observations.

Solution. The minimum is 55,000 and the maximum is 66,000, so the denominator is 66,000 − 55,000 = 11,000.
ID 1: (55,200 − 55,000) ÷ 11,000 = 200 ÷ 11,000 = 0.0182.
ID 2: (60,500 − 55,000) ÷ 11,000 = 5,500 ÷ 11,000 = 0.5000.
ID 3: (66,000 − 55,000) ÷ 11,000 = 11,000 ÷ 11,000 = 1.0000.
ID 6: (55,000 − 55,000) ÷ 11,000 = 0.0000.
Every value now lies between 0 and 1, and the two ends of the range are pinned to exactly 0 and 1 by construction. That is precisely why a single extreme observation would compress everything else into a narrow band near zero.

3. Standardise the same variable.

Solution. First the mean:
(55,200 + 60,500 + 66,000 + 55,000) ÷ 4 = 236,700 ÷ 4 = 59,175.
Then the deviations: −3,975, +1,325, +6,825 and −4,175. Squaring and averaging over the four observations gives 81,567,500 ÷ 4 = 20,391,875, and the square root of that is 4,515.74.
Dividing each deviation by 4,515.74:
ID 1: −3,975 ÷ 4,515.74 = −0.8803.
ID 2: +1,325 ÷ 4,515.74 = +0.2934.
ID 3: +6,825 ÷ 4,515.74 = +1.5114.
ID 6: −4,175 ÷ 4,515.74 = −0.9245.
The standardised values average to zero and have a standard deviation of one. Note that they are not confined to any fixed interval, which is exactly the property that makes standardisation more tolerant of an extreme observation than normalisation.

Example 3 · Worked

Lynn Lee, a data scientist on Wang team at LendALot, pulls a sample of loan records for manual inspection before the team commits to a cleansing plan.

LendALot sample records

Row	ID	Applicant	Product	Score	Income (USD)	Amount advanced (USD)	Outcome
2	1	Mr. Alpha	Mortgage	685	34,000	10,000	No Default
3	2	Ms. Beta	Student Loan	770	−63,050	49,000	No Default
4	3	Mr. Gamma		730	20,565	35,000	Defaulted
5	4	Ms. Delta	Mortgage	664	50,021	unknown	No Default
6	5	Mr. Epsilon	Car Loan	705	100,350	129,000	Defaulted
7	6	Mr. Zeta	Boat Loan	800	800,000	300,000	No Default
8	6	Mr. Zeta	Boat Loan	800	800,000	300,000	No Default

Wang then makes two proposals. Statement 1: keep the ID column and drop the Name column. Statement 2: create a new feature equal to the loan amount expressed as a percentage of income.

1. Classify the defect in each of the rows for Ms. Beta, Mr. Gamma, Ms. Delta and Mr. Zeta.

Solution.
Ms. Beta: invalidity. Income is shown as −63,050. Income cannot be negative, so the value falls outside any meaningful range.
Mr. Gamma: incompleteness. The loan type field is empty.
Ms. Delta: inaccuracy. The loan amount reads unknown. The value is not out of range and nothing contradicts it, but it cannot be true: LendALot necessarily knows how much it advanced to a borrower, particularly one whose loan outcome is recorded as no default. The field simply does not measure the true value.
Mr. Zeta: duplication. Row 8 repeats row 7 in every field.

2. Classify each of Wang two proposals.

Solution. Statement 1 is selection: an entire column is being dropped because it is not needed, and ID alone is sufficient to identify an observation. Statement 2 is extraction: a new variable is being created out of two existing ones, which is what extraction means in the preprocessing vocabulary. It is worth being precise here, because the ratio also looks like something a modeller would call engineering. Within the preparation stage the label is extraction; feature engineering is a distinct activity that belongs to the exploration stage.

Unstructured data are not laid out in any systematic form that a computer can read directly. They exist in formats built for people: text, images, video, audio. Roughly 80% of all data available today is unstructured. Before any of it can train a model it has to become structured, and the version of that process which applies to text is called text processing. Text processing divides, like its structured counterpart, into cleansing and preprocessing. Everything that follows assumes English-language text.

Text cleansing

Raw text is a sequence of characters carrying a good deal that is of no use: markup tags, punctuation, and whitespace of every kind including tabs, line breaks and new lines. A web page that looks clean to a reader is not clean underneath. The home page of a fictional company, Robots Are Us, displays a heading and a single line of copy, but the downloadable source behind it carries markup around each fragment.

Cleansing operations lean heavily on regular expressions. A regular expression, usually shortened to regex, is a sequence of characters in a defined order used to search for a pattern in text. The pattern <.*?> finds every markup tag written in angle brackets, because it matches an opening bracket, then anything of any length, then a closing bracket. GREP, standing for global regular expression print, is the search utility commonly available in programming languages for this purpose. Once a pattern has been located it can be deleted or replaced. Dedicated parsers and packages in R and Python handle the same job at a higher level.

Four operations make up the cleansing step, and the order in which they are applied matters.

Remove markup tags. Text harvested from web pages inherits the markup that surrounded it. A regex identifies each tag and strips it. It is not unusual, though, to retain a few generic tags where they preserve formatting that carries meaning.
Remove punctuation. Most punctuation contributes nothing and goes. Some of it does contribute, however: percentage signs, currency symbols and question marks can matter for training, and rather than deleting them outright they are replaced with word annotations such as /percentSign/, /dollarSign/ and /questionMark/, which preserve the semantic content for later stages. Full stops need particular care because they serve three different purposes: abbreviation, sentence boundary and decimal point. Each use has to be identified and treated on its own terms. As a general rule the full stop after an abbreviation can simply go, while the full stop that ends a sentence is replaced with the annotation /endSentence/. Hyphens and underscores are often retained so that a compound such as e-mail survives as one term.
Remove numbers. Digits are deleted or swapped for the annotation /number/. The annotation tells the computer that a number was present without letting the specific value matter, which is what you want when the value has no bearing on the category being predicted. Left in place, every distinct number would be treated as a separate word, complicating the analysis and adding noise. The exception is any application where the value itself is the output of interest. Information extraction is the leading case: pulling monetary amounts out of financial reports depends entirely on the digits and decimals being kept.
Remove whitespace. Extra spaces, tab spaces and leading or trailing spaces are stripped. Some of them were there to begin with and some are created by the three operations above, which is why this step comes last. The text mining package in R provides a stripwhitespace function for the purpose.

Sequence matters more than it first appears. Consider a passage reporting capital expenditure of $800,000 against $1.2 million a year earlier. Once punctuation is removed, the string 1.2 million becomes 12 million. That is harmless here only because the next operation replaces every number with a /number/ tag, so the corrupted value never survives into the analysis. Had numbers been retained, removing punctuation first would have silently changed the data.

One sentence through the cleansing pipeline

Stage	Text
Raw, after scraping	Profit before taxes was EUR 4.0 mn , down from EUR 4.9 mn .
After cleansing	Profit before taxes was EUR million down from EUR million

Punctuation and digits are gone; the magnitude word million has been preserved deliberately, for reasons covered in the project section.

Tokens, tokenisation and normalisation

A token is the equivalent of a word, and tokenisation is the operation that splits a text into its separate tokens. After tokenisation a text is simply a collection of tokens. Tokenisation can be done at word level or at character level, and word level is by far the more common.

Text then has to be normalised, exactly as structured data had to be scaled. Four operations are involved.

Lowercasing. Folding everything to lower case collapses the artificial distinction between the same word capitalised and uncapitalised, so that The and the are handled as one token.
Removing stop words. Stop words are very common words such as the, is and a. They carry no semantic weight for most text analysis and are dropped to shrink the number of tokens in the training set. Predefined stop word lists ship with the major languages. The judgement is not automatic, though. Advanced applications may need stop words retained in order to read the context of the words next to them. Lists can also be extended for a particular corpus: the word exhibit is not a stop word in general, but in a body of regulatory filings it appears so often that it behaves like one.
Stemming. Stemming converts inflected forms of a word to a base form called the stem. It is rule-based, and the output need not be a real word or the true morphological root. Porter algorithm is the most widely used method. The stem of analyzed and analyzing is analyz; the British variant analysing stems to analys. Implementations exist in R and Python, and the R text mining package exposes a stemDocument function.
Lemmatisation. Lemmatisation converts inflected forms to the true morphological root, the lemma. It is algorithmic and relies on knowledge of the word and of the structure of the language, so the lemma of analyzed and analyzing is the real word analyze. It is more advanced and computationally more expensive than stemming.

Either technique reduces the number of distinct forms in which the same idea appears while keeping the semantic structure of the text. Stemming is more common in English simply because it is easier to perform. Both attack data sparseness, the condition in which a corpus is dominated by unique, very low frequency tokens. By collapsing many rarely occurring words into fewer, denser stems or lemmas, both techniques allow a less complex model to be trained.

Bag-of-words and the document term matrix

Once the cleansed text has been normalised, a bag-of-words is built. The BOW is the set of distinct tokens found across every text in the sample. It is a set, so it records which tokens exist and nothing about where they sat or in what order. That is a real limitation, and it is also why the representation is memory efficient and easy to work with.

Consider four short cleansed texts: the man went to the market today; market values are increasing; increased marketing is needed; there is no market for the product. Tokenising them and then normalising step by step shows the BOW shrinking at every stage. Before normalisation the BOW holds every distinct form, including The and Market with initial capitals. Lowercasing merges those with their lower-case twins. Removing stop words strips out the, to, is, are, there, no and for. Stemming then collapses values and value into valu, and increasing and increased into increas.

The bag-of-words shrinking through normalisation

Stage	Resulting bag-of-words
Before normalising	the, man, went, to, market, today, Market, values, are, increasing, Increased, marketing, is, needed, There, no, for, product, The
After lowercasing	the, man, went, to, market, today, values, are, increasing, increased, marketing, is, needed, there, no, for, product
After removing stop words	man, went, market, today, values, increasing, increased, marketing, needed, product
After stemming	man, went, market, today, valu, increas, need, product

The final BOW then becomes the column set of a document term matrix. The DTM is to text what a data table is to structured data. Each row is one document, so the number of rows equals the number of documents in the sample. Each column is one token, so the number of columns equals the size of the BOW built across all those documents. The cells can hold counts of how often each token appears in each document, and later in this lesson they will hold other frequency measures instead. At the moment the DTM is populated, the unstructured text has become structured data.

Document term matrix for the four texts, cells filled with counts

	man	went	market	today	valu	increas	need	product
Text 1	1	1	1	1	0	0	0	0
Text 2	0	0	1	0	1	1	0	0
Text 3	0	0	1	0	0	1	1	0
Text 4	0	0	1	0	0	0	0	1

N-grams

The weakness of the bag-of-words is visible in the fourth text. The phrase reads there is no market for the product, which is plainly negative, yet the token no was removed as a stop word and the surviving row is indistinguishable from a positive statement about a market. Sequence has been destroyed along with position.

N-grams repair this by representing sequences of words. The length of the sequence runs from 1 to n: one word is a unigram, a two-word sequence is a bigram, a three-word sequence is a trigram, and so on. The BOW shown above is a unigram BOW. N-grams are used in exactly the same way as unigrams to build a BOW, and different lengths are routinely combined in one BOW before a DTM is constructed. Stemming can be applied to the cleansed text before n-grams are formed.

N-grams from the clean text: the man went to the market today

Type	Tokens produced
Unigrams	The, man, went, to, the, market, today
Bigrams	The_man, man_went, went_to, to_the, the_market, market_today
Trigrams	The_man_went, man_went_to, went_to_the, to_the_market, the_market_today

Normalisation behaves differently once n-grams are in play. Stop words persist when they are welded to a neighbour: the bigram to_the consists of two stop words but will not be matched by a stop word list, which contains only isolated words. Removing stop words from an n-gram BOW is therefore a weaker operation than it is for unigrams, and that is not always a bad thing, since it is exactly what keeps a negation such as no_longer intact.

Example 4 · Worked

Answer the following on unstructured data preparation and wrangling.

1. What is the output of preparing and wrangling textual data?

Solution. A document term matrix. The whole objective of the stage is to convert unstructured input into structured output, and the DTM is the structured object that results. It is the text equivalent of a data table and can be read directly by a computer. A confusion matrix is a performance evaluation device and belongs much later in the process.

2. Which cleansing operation may require an annotation to be added?

Solution. Removing punctuation. Percentage signs, currency symbols and question marks can be informative for training, so when they are stripped an annotation is substituted to keep the meaning. Removing markup tags and removing whitespace destroy nothing worth keeping and need no annotation.

3. What does a column of a document term matrix represent, and what does a cell contain?

Solution. A column represents a token drawn from the bag-of-words built across all documents in the sample. A cell contains a count of tokens, specifically the number of occurrences in that document of the token named in the column heading. A row, by contrast, is one document, and the row count is therefore the document count.

4. Which of the following belongs to normalisation rather than cleansing: removing numbers, removing whitespace, lowercasing the alphabet?

Solution. Lowercasing the alphabet. Removing numbers and removing whitespace are both cleansing operations, performed before any token is formed. Normalisation acts on tokens and consists of lowercasing, stop word removal, stemming and lemmatisation.

5. A corpus is dominated by words that appear only once or twice. Which technique reduces the risk of training an over-complex model: stemming, scaling or data cleansing?

Solution. Stemming. The condition described is data sparseness. Stemming maps many inflected forms onto a single base form, so a large number of very rare tokens collapses into a smaller number of denser ones. Fewer features means less complexity and less scope for overfitting. Scaling applies to numeric features in structured data and does not touch this problem; cleansing removes defects but does not consolidate word forms.

6. Which statement about tokenisation is accurate: it is part of cleansing, it is most commonly performed at character level, or it splits a text into separate tokens?

Solution. It splits a text into separate tokens, which is the definition. It is not part of cleansing, since it operates on already cleansed text and opens the preprocessing stage. It is most commonly performed at word level, not character level.

Exploration sits between preparation and training. Its purpose is to investigate and understand the distributions and relationships inside the prepared data, and what is learned here is used for the rest of the project. The quality of exploration bears directly on the quality of the trained model. Domain knowledge is again essential, because exploration conducted without it readily produces spurious relationships that mislead every step that follows. The stage should involve analysts, model designers and subject specialists together.

Three tasks make up exploration. Exploratory data analysis is the preliminary survey. Feature selection keeps only the pertinent features for training, which lowers model complexity and shortens training time. Feature engineering creates new features by transforming or recombining existing ones. Model performance depends heavily on the last two.

Exploratory data analysis

EDA uses graphs, charts and other visualisations, including heat maps and word clouds, to summarise and observe the data. Many exploratory graphs are made quickly and thrown away, which is the point: statistical programming environments and ordinary spreadsheet tools produce them fast. Quantitative summaries such as descriptive statistics and measures of central tendency do the same job in numbers.

One objective of EDA deserves emphasis because it is easy to overlook. EDA is a communication medium among the people with a stake in the project: business users, domain experts and analysts. Quick visualisations let that group connect and confirm that the prepared data make sense. The other objectives are to understand the properties of the data, to find patterns and relationships, to test basic questions and hypotheses, to document distributions and other characteristics, and to plan the modelling strategy for the steps ahead.

Technique follows dimensionality. For high-dimension data with many features, a dimension reduction technique such as principal components analysis makes exploration tractable in the first place.

One dimension. Summary statistics for a single feature: mean, median, quartiles, ranges, standard deviations, skewness and kurtosis. The visual counterparts are histograms, bar charts, box plots and density plots. Histograms show equal bins of data with their frequencies and give a high-level view of the distribution. Bar charts summarise the frequencies of categorical variables. Box plots display the distribution of continuous data by picking out the median, quartiles and outliers of a normally distributed feature. Density plots are smoothed histograms and are often laid over the histogram itself. A histogram of hypothetical entry-level analyst salaries at United Kingdom banks, for instance, shows a normal distribution with an approximate mean of £68,500.
Two dimensions. A correlation matrix summarises relationships numerically. Scatterplots and line graphs do it visually. Overlaying one-dimensional plots allows features to be compared, and colour, shape, size and legends can be used to pack more information into fewer charts.
Many dimensions. Stacked bar and line charts, multiple box plots and multi-coloured or multi-shaped scatterplots. Arranging several box plots in one chart, each representing a feature on the x-axis against a target variable on the y-axis, assesses the relationship between each feature and the target at a glance.

Two worked illustrations from the curriculum are worth carrying. A multi-box plot chart of units of shares purchased against stock price, with price shown in increments of $0.125, reveals that the median number of shares purchased is highest at a price of $0.25 and lowest at $0.625, while also making clear that the differences across price points are not visually large. A scatterplot of age against annual salary shows salary tending to rise with age, a pattern that looks real to the eye but which may not be statistically significant.

That caveat generalises. A scatterplot is a starting point, not a conclusion. Any apparent relationship has to be tested formally. Common parametric tests are ANOVA, the t-test and the Pearson correlation; common non-parametric tests are the chi-square test and the Spearman rank-order correlation. Alongside visualisation, descriptive statistics remain a good summary tool: central tendency plus minimum and maximum for continuous variables, counts and frequencies for categorical ones.

EDA is not only diagnostic. The relationships and trends it exposes suggest new features, which is where exploration hands over to engineering.

Feature selection

Structured data present themselves as features, one per column. Once EDA has revealed the relevant patterns, unneeded, irrelevant and redundant features have to be identified and removed. Basic diagnostic testing should look for redundancy, heteroscedasticity and multi-collinearity. The objective is to find the features that preserve the important patterns and complexities of the full dataset while requiring less data overall, which matters because computing power costs money and time.

Even after preparation, structured data typically retain features that add nothing to accuracy or that actively degrade training. What is wanted is a parsimonious model: as few features as possible, with the greatest out-of-sample predictive power those features can deliver.

Do not confuse feature selection with the preprocessing steps of the preparation stage. Preprocessing needs no more than clarification from data administrators and basic intuition, such as knowing that salary and income overlap. Feature selection needs an understanding of the data and of statistics, and it depends on comprehensive EDA having been done first.

The process is methodical and iterative. Statistical measures assign a score reflecting the importance of each feature, features are ranked on that score, and each is then retained or discarded. The measures used are generally univariate, considering one feature at a time either on its own or against the target variable. The standard choices are the chi-square test, correlation coefficients and information-gain measures such as R-squared values from a regression. These can be combined so that each method is applied to each feature, with automatic backward and forward passes over the feature set to refine the selection. Prebuilt functions for all of this exist in the usual modelling languages.

Dimensionality reduction is related but distinct. It identifies the features that account for the greatest variance between observations and lets a smaller volume of data be processed, cutting memory requirements and speeding up learning algorithms. The difference from feature selection is fundamental: dimensionality reduction builds new, uncorrelated combinations of features, whereas feature selection includes or excludes the original features without altering any of them. Both shrink the feature count; only one changes what the features are.

Feature engineering

Once the right features have been selected, engineering optimises them. How well a model trains depends on how well the data are presented to it, and engineering aims to produce features that describe the structures inherent in the dataset. What counts as a good feature depends on the project, the domain and the nature of the problem. In practice engineering either transforms an existing feature into a new one or decomposes it into several.

Continuous data. A new feature might be the logarithm of the product of two or more existing features. A more instructive case is income. Different salary bands attract different tax rates, so domain knowledge can decompose an income feature into bands, producing a new feature named income_above_100k taking the values 0 and 1, where 1 records an annual salary above $100,000. Grouping subjects this way lets assumptions about income tax be built into a model that uses the tax consequences of higher and lower salaries to make financial predictions.
Categorical data. A new feature can combine two features, by sum or product, or split one into many. A single education level feature with five possible values, high school, associates, bachelors, masters and doctorate, can be decomposed into five new features, one per value, such as is_highSchool and is_doctorate, each filled with 0 for false and 1 for true. Converting categorical variables into binary form for machine reading in this way is called one hot encoding, and it is among the most common methods for handling categorical features in text data.
Date and time data. A timestamp can be decomposed into several engineered features, naming for instance the second within the hour, the hour within the day, and the day component of the date. Attributes of this kind matter a great deal in modelling trading algorithms.

Engineering systematically alters, decomposes or combines existing features to produce more meaningful ones, and more meaningful features let a model train faster and more easily. It is worth appreciating how much leverage sits here: different engineering strategies applied to the same underlying data and the same algorithm can generate dramatically different results.

The same three tasks apply to text, but the objects being explored are tokens rather than columns of numbers, and the statistics used are frequency measures.

Exploratory analysis of text

Text analytics has four common applications. Text classification sorts texts into classes using supervised methods. Topic modelling groups texts into topic clusters using unsupervised methods. Fraud detection is a third. Sentiment analysis predicts whether a text is negative, neutral or positive, and uses both supervised and unsupervised approaches.

A collection of texts is a corpus, and each text in it is a sequence of tokens. Exploratory analysis of a corpus computes basic statistics on those tokens. The central one is term frequency, the number of times a token occurs across every text in the dataset, expressed as a ratio to the count of all tokens the dataset contains. Alongside it sit word associations, mean word length, mean sentence length, and counts of words and of syllables.

Which statistics matter depends on the application. In topic modelling, the most informative words are found by computing the term frequency of each word, and the words with high TF values are then eliminated, because a token that appears everywhere is either a stop word or ordinary vocabulary. Removing them leaves a compact BOW whose contents genuinely relate to the topics present. The word soccer, for instance, is informative for the topic sports precisely because it does not appear in most texts. In sentiment analysis and text classification, the chi-square measure of word association is more useful, since it shows which words appear significantly in the negative as against the positive sentences, or in one class of document as against another.

Visualisation uses the same devices as for structured data. Bar charts show word counts or frequencies. Word clouds are the characteristic text visualisation: the most frequent words in a corpus are displayed with font size proportional to frequency, and colour adds a further dimension such as word length. A word cloud is the fastest way to see what a body of text is about.

Selecting features from text

Feature selection for text means choosing a subset of the tokens in the dataset, since the tokens are the features. It shrinks the vocabulary, which makes the model more efficient and less complex, and it removes noisy features that do not help training and may actively damage accuracy.

Noisy features live at both ends of the frequency distribution, and it is worth being clear about why each end is harmful, because the two failures are opposite in kind.

The most frequent tokens are typically stop words present in nearly every text. They strain the model when it tries to draw a decision boundary, because a term shared by all texts cannot separate them. That is a route to underfitting.
The most sparse tokens appear in only a handful of texts. They tempt the model into assigning any text containing such a term to one specific class, on evidence that is far too thin. That is a route to overfitting.

Text classification means sorting documents into assigned classes, where a class is simply a category such as relevant against irrelevant, or bearish against bullish. Removing noise features is critical for it. Three methods do the work.

Frequency measures. Vocabulary pruning filters out tokens with very high and very low TF values across the corpus. Document frequency is a second frequency measure, defined as the number of documents containing a token divided by the total number of documents. It discards features that carry no information about the class of a text because they are present in all of them. DF is the simplest selection method available and often performs well when the token count runs to many thousands.
Chi-square test. Applied here, the test examines whether two events are independent: the occurrence of a token and the occurrence of a class. It ranks tokens by their usefulness to each class. Tokens with the highest chi-square statistics occur disproportionately in texts of one class, so they have the greatest power to discriminate and are the ones selected as features.
Mutual information. MI measures how much information a token contributes to a class of texts. Its value is 0 when the token is distributed identically across every class, since a token that appears equally everywhere tells you nothing. It tends towards 1 as the token concentrates in a single class.

Mutual information for tokens in the sports class

Token	MI value
soccer	0.0781
cup	0.0525
match	0.0456
play	0.0387
game	0.0299
team	0.0265
win	0.0189

Mutual information for tokens in the politics class

Token	MI value
election	0.0612
president	0.0511
polls	0.0341
vote	0.0288
party	0.0202
candidate	0.0201
campaign	0.0201

Note how tightly the highest-scoring tokens track the name of their own class, and how quickly the values fall away as the tokens become more generic.

Engineering features from text

Engineering for text overlaps with processing but goes further, and it remains a mixture of art and science. Four techniques matter.

Numbers. Processing turns every number into a single token such as /number/. That throws away a real signal, because numbers of different lengths mean different things. A four-digit number is likely a year; a long string of digits is likely an identifier. Replacing four-digit numbers with /number4/ and ten-digit numbers with /number10/ preserves that distinction at no cost.
N-grams. Multi-word patterns that discriminate well are identified and kept intact. The word market on its own is common and could point to any number of subjects, whereas the pair stock market is used in a specific context and helps separate finance-related text from general text. Treating the two adjacent words as the single token stock_market is what captures that.
Name entity recognition. NER is available as a library or package in most languages. The algorithm examines each token together with the semantics around it, consults its dictionary, and tags the token with an object class. Object classes include ORGANIZATION, DATE, LOCATION, and also MONEY, TIME and PERCENT. The tags can be used directly as features. They also serve a second purpose: identifying tokens on which lowercasing or stemming should be suppressed, because in a phrase such as CFA Institute the word Institute is part of an organisation name rather than a common noun. Suppressing normalisation there makes the feature more discriminative, not less.
Parts of speech. POS tagging likewise uses language structure and dictionaries to label every token with its part of speech: noun, verb, adjective, proper noun and the rest. The tags can be used as features and can be counted to describe the composition of a text. A text dense in proper nouns probably concerns people and organisations and may be a business topic. POS tags also disambiguate: market used as a verb, as in to market a product, suggests the text is about marketing, whereas market used as a noun suggests a physical or stock market and possibly trading. Compound nouns such as CFA Institute can be treated as one token for tagging purposes.

NER and POS tags on the sentence: CFA Institute was formed in 1947 and is headquartered in Virginia

Token	NER tag	POS tag	Meaning of the POS tag
CFA	ORGANIZATION	NNP	Proper noun
Institute	ORGANIZATION	NNP	Proper noun
formed		VBN	Past participle
1947	DATE	CD	Cardinal number
headquartered		VBN	Past participle
Virginia	LOCATION	NNP	Proper noun

Tokens carrying no NER tag, such as was, in, and and is, still receive POS tags: past-tense verb, preposition, coordinating conjunction and third-person singular present verb respectively.

Many further techniques exist. The unifying goal of all of them is to keep the semantic essence of the text while simplifying it and converting it into structured data.

Example 5 · Worked

Eric Kim on Wang team at LendALot is asked to explore the available data and report on patterns, trends and interesting variables. The sample he pulls carries the structured fields alongside borrower free-response text explaining their credit score. Two borrowers who did not default wrote that they have worked hard to improve or maintain their position; a borrower who defaulted wrote that the score would probably have been higher if he had worked harder.

1. Are histograms, box plots and scatterplots relevant here?

Solution. Yes, for the structured part of the sample. The structured features are ID, loan outcome, income, loan amount and credit score. Histograms and box plots would show how income, loan amount and credit score are distributed, and each can be produced both across all historical borrowing instances and separately within the defaulted and non-defaulted subsets, which is where the interesting comparison lies. Scatterplots of income against loan amount, income against credit score, and loan amount against credit score, again both overall and within each outcome group, would expose relationships among the continuous variables. None of these tools applies to the free-response column.

2. Name a visualisation technique for the free responses.

Solution. A word cloud. It gives an immediate view of the most frequently occurring words, which is term frequency shown graphically. Some of what surfaces will be obvious and of limited value, such as credit and score. Other frequent words are more promising for creditworthiness prediction: worked, hard, probably, embarrassed, regret, good, decent and great all describe attitude rather than arithmetic.

3. How can ranking methods be used to pick out interesting features?

Solution. Kim can score and rank every feature. The target variable, loan outcome, is discrete, so chi-square and information gain are well suited. Both are univariate, scoring each feature variable on its own. Importantly, they are not restricted to the structured columns: the same ranking can be applied to word-count features such as term frequency and document frequency derived from the frequently occurring words the word cloud identified, so text-derived features compete on the same footing as income and credit score.

4. Give an example of a bigram from the free-response text that might discriminate between loan outcomes.

Solution. credit_score and worked_hard both have potential. The second is the more interesting of the two, for a reason the next example develops: the unigram worked appears in both defaulted and non-defaulted texts and so looks uninformative on its own, whereas the sequence around it is not the same in the two groups.

Example 6 · Worked

Wang now asks Lee and Kim to propose textual feature representations for the creditworthiness task, working from the same sample of six borrower records.

1. Which three representations should they consider, and which should they reject?

Solution. They should consider bag-of-words, n-grams and parts of speech. They should set aside name entity recognition, because the prospective borrower text contains no explicit references to people, locations, dates or organisations, so the NER tagger would have nothing to tag.

2. Justify each of the three.

Solution. Take them in turn.

Bag-of-words. BOW applies in almost any context where token boundaries are explicit in the language, as in English, or can be inferred by processing, as in Spanish. It is the natural starting point for any project exploring text feature representations.

N-grams. Sequences add contextual information that enriches a BOW. The concrete case is the token worked, which appears three times across the sample, twice in no-default texts and once in a defaulted text. On that evidence alone the token looks like it refers to work ethic and might predict creditworthiness. Looking instead at trigrams containing it separates the cases cleanly: have_worked_hard appears in the two no-default texts, describing what the borrower has achieved and plans to do, while had_worked_harder appears in the defaulted text, describing what was not done. The unigram cannot distinguish those two meanings; the trigram can.

Parts of speech. POS tags describe the composition of each text. They reveal whether a borrower is writing in action words, meaning verbs, or in descriptors, meaning adjectives, and whether that composition differs systematically between the no-default and defaulted groups.

Model training is systematic, iterative and recursive. How many iterations are needed before results are good enough depends on two things: the nature of the problem and of the input data, and the level of performance the practical application actually demands. There is no fixed answer and no standard rulebook, but a working knowledge of the training data in its domain and of the principles behind the algorithm makes the difference between disciplined training and guesswork. Machine learning engineers and domain experts have to build the model together.

Three tasks make up the stage. Method selection decides which algorithm or algorithms to use, guided by the classification task, the type of data and the size of the data. Performance evaluation applies a set of complementary techniques and measures to quantify how well the model does. Tuning is the set of decisions and actions taken to improve it. The three repeat until performance is acceptable. As with the earlier stages, nothing should start until the problem has been stated, objectives defined, useful data points identified and the model conceptualised, and because the process is iterative that plan will be revised repeatedly as it evolves.

Structured and unstructured data train the same way

The training process itself does not differ between the two. Almost all algorithms expect structured input, which is why the preparation stage exists at all: unstructured material is processed and organised into structured form before it arrives here. Text has been shown becoming a data matrix; other unstructured forms are prepared into matrices or tables in the same spirit.

Evaluating the fit of an algorithm

The fundamental idea of training is to fit a system of rules to a training dataset so that a pattern in the data is revealed. Fitting describes how well the resulting model generalises to new data. A good fit produces good performance and can be validated on data outside the training set, that is, out of sample. Three outcomes are possible, and they are best seen as decision boundaries drawn through a classification problem with two classes of observation.

Underfit. The boundary is too simple to follow the data. In the standard illustration the underfit model misclassifies four observations, three of one class and one of the other. It has not learned enough.
Overfit. The boundary contorts itself to capture every training observation and achieves perfect accuracy in sample, with no errors at all. It has learned the training set rather than the pattern, and it is therefore unlikely to perform on future cases.
Good fit. The boundary follows the data closely but not slavishly. In the illustration it makes one classification error. It fits the training data well without fitting them so well that it cannot generalise.

Notice what that means for how a model should be judged: the best in-sample accuracy belongs to the model you least want.

Figure 3: Three fitting scenarios

Circles and squares are the two classes; the accent line is the decision boundary. The middle panel is the objective, even though the right-hand panel scores better on the training data.

What causes a bad fit

Two properties of the dataset drive most fitting errors.

Size of the dataset. A small dataset tends to cause underfitting, because a small sample often does not contain enough evidence for the pattern to show itself. Constrained this way, the algorithm cannot recognise relationships that are genuinely there.
Number of features. Too few features also cause underfitting, for the same reason: a narrow feature set may not carry the characteristics that explain the link between the target and the predictors. Too many features cause the opposite problem. A large feature set complicates the model and can distort the patterns in the data through low degrees of freedom, which is overfitting. This is why feature selection, using the chi-square and mutual information techniques described earlier, is a key defence against overfitting.

Feature engineering works on the other side of the trade. Well-engineered new features raise the underlying data points that better explain how features interact, so engineering tends to prevent underfitting. Selection guards against overfitting; engineering guards against underfitting. That pairing is worth memorising, because it is the crispest single statement of what the exploration stage contributes to model quality.

Method selection

Choosing an algorithm is craft rather than procedure, with no strict guidelines, and it is the first step of training. Three factors govern it.

Supervised or unsupervised. Supervised datasets contain ground truth, the known outcome for each observation. Unsupervised modelling is harder precisely because there is no ground truth and no target variable. The distinction cuts deeper than data availability: a supervised model imposes a structure that the data may or may not support, whereas an unsupervised model brings no structure beyond whatever emerges from the data themselves. For supervised learning with labelled training data the usual candidates are regression, ensemble trees, support vector machines and neural networks; predicting default from high-yield corporate bond issuer data is a typical application. For unsupervised learning the usual candidates are dimensionality reduction, clustering and anomaly detection; grouping financial institutions by their financial attributes is a typical application.
Type of data. Numerical data, such as predicting share prices from historical market values, suit classification and regression tree methods. Text data, such as predicting the topic of a financial news article from its headline, suit generalised linear models and support vector machines. Image data, such as identifying tanker ships moving in and out of port in a satellite image, suit neural networks and deep learning. Speech data, such as predicting financial sentiment from quarterly earnings conference call recordings, also respond well to deep learning.
Size of data. A dataset is described by two numbers, the count of instances and the count of features, and their combination steers the choice. Support vector machines have been found to work well on wider datasets carrying 10,000 to 100,000 features with relatively few instances. Neural networks tend to work better on longer datasets, where instances greatly outnumber features.

Once a method is chosen, method-specific decisions follow, such as the number of hidden layers in a neural network or the number of trees in an ensemble. Real datasets often mix numeric and text material, and mixed data can be handled by combining the results of more than one method. Sometimes the prediction of one method becomes a feature for another: unstructured financial text can be run through a logistic regression to classify stock sentiment as positive or negative, and that classification can then enter a larger CART model that also takes structured financial data as predictors for stock selection. More than one method can also be run in parallel with the outputs combined by quantitative or subjective weights, so that the advantages of each are exploited.

Splitting the data

Before supervised training begins, the master dataset is divided into three parts.

A training set, used to fit the model, taking roughly 60% of the master dataset.
A cross-validation set, also called the validation set, used to tune and validate, taking roughly 20%.
A test set, used to test the finished model, taking what remains.

The split is made by random sampling, using a technique such as the k-fold method. The 60:20:20 ratio is the common recommendation, though the percentages can vary. Unsupervised learning needs no split at all, because there are no labels to hold back.

Class imbalance

Supervised classification runs into trouble when one class vastly outnumbers the others, because the objective of the algorithm is to maximise accuracy. Take default prediction for high-yield corporate issuers rated in the BB+/Ba1 to B+/B1 range. Issuers that defaulted, the positive or class 1 cases, are very few beside issuers that did not, the negative or class 0 cases. On such data a naive model that predicts no issuer will ever default can post a high accuracy while misclassifying every single default. The metric looks excellent and the model is worthless.

Balancing the training data addresses this. The majority class can be randomly undersampled, or the minority class randomly oversampled. Sampling may be done with or without replacement, since the two are equivalent in general probability theory. Which of undersampling and oversampling to use depends on the specific problem. More advanced techniques generate synthetic observations from the existing data and add them to the dataset to fill out the minority class.

Measuring training performance, or goodness of fit, is what validates a model. Three techniques are used, and the first two are built for binary classification.

Error analysis and the confusion matrix

For a classification problem, error analysis computes four counts. A true positive is a class 1 case correctly predicted as class 1. A true negative is a class 0 case correctly predicted as class 0. A false positive is a class 0 case wrongly predicted as class 1, and is also called a Type I error. A false negative is a class 1 case wrongly predicted as class 0, and is also called a Type II error. The four are arranged in a grid called the confusion matrix, with predicted results down one side and actual training labels across the top.

Two further metrics come straight out of those counts, and the way to remember which is which is to ask what sits in the denominator. Take a quality inspection setting in which class 0 means not defective and class 1 means defective.

Correctly predicted positives as a share of all predicted positives.

Precision matters when a false positive is expensive. An expensive product fails inspection, is predicted class 1 and is scrapped, when in fact it was perfectly good and belonged in class 0. The cost of that mistake is the whole product.

Correctly predicted positives as a share of all actual positives. Recall is also called sensitivity.

Recall matters when a false negative is expensive. An expensive product passes inspection, is predicted class 0 and is shipped to a valued customer, when it was in fact seriously defective and belonged in class 1. The cost of that mistake is the customer.

Trading one against the other is a business decision, not a statistical one, which is why two overall measures are also needed.

The percentage of all predictions, across both classes, that were correct.

The harmonic mean of precision and recall.

High scores on both suggest a good model. The reason F1 is the harmonic mean and not the arithmetic mean is that the harmonic mean is pulled towards the smaller of the two inputs, so a model that achieves a high score on one measure by sacrificing the other cannot hide behind the average. F1 is more appropriate than accuracy whenever the class distribution is unequal and the balance between precision and recall has to be measured. Where the classes are equal in number, accuracy is a perfectly appropriate headline measure.

Example 7 · Worked

A model is run over ten observations. The actual labels and the predicted results give three true positives, one false positive, two false negatives and four true negatives.

Confusion matrix for the ten-observation sample

Predicted	Actual class 1	Actual class 0
Class 1	3 (TP)	1 (FP)
Class 0	2 (FN)	4 (TN)

1. Compute precision, recall, F1 score and accuracy.

Solution.
Precision = 3 ÷ (3 + 1) = 0.75.
Recall = 3 ÷ (3 + 2) = 0.60.
F1 score = (2 × 0.75 × 0.60) ÷ (0.75 + 0.60) = 0.90 ÷ 1.35 = 0.67.
Accuracy = (3 + 4) ÷ (3 + 1 + 4 + 2) = 7 ÷ 10 = 0.70.

2. Interpret the results.

Solution. Had every class 1 case been predicted correctly there would be no false positives and precision would equal 1. Had every class 0 case been predicted correctly there would be no false negatives and recall would equal 1. Either way F1 would then equal 1 as well. As it is, precision of 0.75 against recall of 0.60 says the model is better at avoiding false positives than at avoiding false negatives. The F1 score of 0.67 finds the balance between the two, and note where it lands: closer to 0.60 than to 0.75, that is, nearer the weaker of the two figures. The arithmetic mean would have been 0.675, which flatters the model slightly. Accuracy is 0.70, and it would be 1 if every prediction were correct. Because this dataset contains equal numbers of class 1 and class 0 cases, accuracy is a reasonable overall measure here. Had the classes been unequal, F1 would have to be used instead.

Receiver operating characteristic

The second technique plots the trade-off between the false positive rate on the x-axis and the true positive rate on the y-axis, as the cutoff point is varied. In a logistic regression the cutoff is the threshold applied to the predicted probability p: if p exceeds the threshold, the observation is classified as class 1, otherwise as class 0.

The true positive rate is the same quantity as recall.

The shape of the curve tells you how good the model is. The more convex the curve, the better the performance. Area under the curve, or AUC, converts the shape into a single number. An AUC close to 1.0 indicates near perfect prediction; an AUC of 0.5 is the diagonal, which is what random guessing produces. A model whose ROC curve bulges hard towards the top left corner, with an AUC above 0.9, is comfortably the strongest of a set of candidates.

Figure 4: ROC curves and area under the curve

The dashed diagonal is the no-information benchmark. Every point on a curve corresponds to one choice of cutoff, so choosing a threshold is choosing a point along the curve.

Root mean squared error

The third technique applies where the prediction is continuous rather than categorical, so it belongs mainly to regression methods. RMSE is a single figure that captures all the prediction errors across n observations: take the difference between each predicted and actual value, square it, average the squares, and take the square root. A small RMSE points to better model performance.

Squaring before averaging means large individual errors dominate the measure.

Evaluation produces findings; tuning acts on them. Two diagnostic readings start the process. A high error rate on the training data by itself says the model has underfitted. An error rate on the cross-validation data that sits materially above the training figure says instead that it has overfitted. Note that the second diagnosis depends on comparing two numbers, which is why the data had to be split before training began.

Fitting error decomposes into two kinds. Bias error goes with underfitting and is high when a model has been simplified to the point where it does not learn enough from the patterns in the training data. Variance error goes with overfitting and is high when a model is so complicated that it memorises the training data and will therefore do badly on anything new. Neither can be eliminated. Both can be minimised together so that their sum, the total aggregate error, is as small as possible, and finding that balance is the bias–variance trade-off.

Parameters and hyperparameters

The distinction matters because only one of the two is tuned.

Parameters are learned from the training data during training itself, by an optimisation technique. They depend on the data. Regression coefficients, the weights in a neural network and the support vectors in a support vector machine are all parameters.
Hyperparameters are used to estimate the parameters and do not depend on the training data. They are set and adjusted by hand. Examples include the regularisation term in a supervised model, the activation function and number of hidden layers in a neural network, the number of trees and the tree depth in ensemble methods, k in k-nearest neighbour classification and k-means clustering, and the p-threshold in a logistic regression.

The p-threshold makes the concept concrete. A researcher classifying sentences from financial statements into positive or negative stock sentiment might begin at a threshold of 0.50, so that any sentence whose modelled probability exceeds 50% is classified positive. A confusion matrix built from running the cross-validation set through that model yields accuracy and F1. The researcher then varies the threshold, to 0.55, 0.60, perhaps 0.65, re-runs the cross-validation set at each value, builds a new confusion matrix each time and compares the resulting scores. The threshold finally chosen is the one whose classifications produce the highest accuracy and F1 score.

There is no formula for estimating a hyperparameter. Tuning heuristics are used, and so is grid search, a method of training the model systematically across many combinations of hyperparameter values, cross-validating each one, and identifying the combination that performs best. The test of an optimal set of values is not raw performance but consistency: the training error and the cross-validation error should be close to one another. That closeness is what indicates the model will generalise to the test set and to genuinely new data rather than overfit.

Fitting curves

Plotting training error against each value of a hyperparameter, which amounts to plotting it against model complexity, produces a fitting curve. Plotting the cross-validation error on the same axes makes the trade-off visible and is the practical tool for tuning. Take the regularisation hyperparameter as the example.

Figure 5: Fitting curve for the regularisation hyperparameter

The gap between the two curves is variance; the height of the training curve is bias. The optimum is where the two are close and both are low, not where either alone is lowest.

Read the diagram from left to right.

Slight regularisation penalises complexity lightly, so most or all features stay in the model and it is free to memorise the data. The training error is small while the cross-validation error is much larger. That difference is variance. High variance error with low bias error means the model performs well in sample but throws off many false positives and false negatives on the cross-validation set. It is overfitted and does not generalise.
Large regularisation penalises complexity excessively, letting too few features into the model so that it learns too little. It may lack the predictor variables and the complexity needed to see the underlying patterns. Both errors are now large, and a large training error is the signature of high bias, which is underfitting. The model performs badly on training and cross-validation data alike, typically because important predictors have been squeezed out.
Optimum regularisation minimises both errors together. It penalises complexity just enough that only the most important features survive, which stops the model memorising while still letting it learn enough to distinguish real patterns. Training error and cross-validation error end up similar and both minimal. The range of optimal values is found heuristically, by grid search among others.

When tuning is not enough

If high bias or high variance persists after the hyperparameters have been tuned, the problem lies with the data rather than the settings. More training instances may be needed. Alternatively the feature count must change: decrease it where variance is high, increase it where bias is high. The model is then retrained and retuned on the revised dataset.

Where the model is complex, in the sense of a large model built from sub-models, ceiling analysis is available. Ceiling analysis systematically evaluates each component in the model-building pipeline to establish which part could be improved by further tuning. A stock market prediction model, for instance, might need both historical market data and news articles about the stocks. A sub-model extracts the relevant information from the articles, or classifies their sentiment, and feeds the result into the larger model as a feature. The performance of the larger model therefore depends on the performance of that sub-model, and ceiling analysis identifies which sub-model to tune in order to lift overall accuracy.

Everything above now runs end to end on one real project. Robo-readers are automated programs that analyse large quantities of text, news articles and social media among them. Investors use them to examine how the views expressed in text relate to future company performance. The dimension they most often look for is sentiment polarity: how positive, negative or neutral a phrase or statement is with respect to a target. In the statement that XYZ Corporation is doing terrific things with its new product innovation, the polarity is positive and the target of that polarity is XYZ Corporation. Sentiment of this kind can carry predictive power both on its own and alongside structured financial data, for individual firms and for portfolios.

The project examines how well sentiment can be classified in English-language news articles on LexisNexis relating to every company listed on the NASDAQ OMX Helsinki in Finland. It follows the text pipeline exactly as set out earlier in this lesson.

Text curation

The text is a collection of English sentences from financial and economic news sources, taken from the Financial Phrase Bank hosted on Researchgate.net. The compressed folder holds six files: a licence file, a readme file, and four files of text data in plain .txt format that any text editor can open. The data are cross-sectional, not a time series, which matters because it removes any question of temporal ordering from the sampling.

The four files hold 14,780 sentences in total, each already labelled with one of three sentiment classes: positive, neutral or negative. The labels are assigned from an investor perspective, so they are intended to indicate whether a sentence would have a positive, neutral or negative influence on the relevant company share price. This project draws on two of the four files, Sentences_AllAgree and Sentences_75Agree, and uses only the sentences labelled positive or negative. That gives 2,180 sentences, of which 1,457 carry the positive label and 723 carry the negative label. A supervised model is trained, validated and tested on those, and can then predict the sentiment class of sentences drawn from similar financial news.

In the raw files each sentence carries its sentiment annotation at the end, introduced by an @ character. That separator is what allows the raw text to be split into a two-column data table, one column for the sentence and one for the sentiment class. A collection of text in any form, whether list, matrix or data table, is a corpus.

Text preparation

Three cleansing decisions were taken, and each is a judgement call rather than a default.

Markup tags. No operation was needed, because the source files contain none.
Punctuation. Percentage and currency symbols were replaced with word annotations before the rest of the punctuation was stripped, so that the presence of a percentage or a currency amount survives into the analysis. Because the sentences had already been identified and extracted from the source, the punctuation that would normally mark sentence boundaries, meaning full stops, semi-colons and commas, was simply removed rather than annotated. Special characters such as the plus sign and the copyright symbol went too. The order is deliberate: word annotations are substituted first, and only then is the remaining punctuation cleared.
Numbers. All numbers were removed. The reasoning is specific to sentiment: the polarity of a sentence depends on its words, not on its figures. In a sentence reporting that a Lithuanian brewer boosted sales last year by 22.3 percent to 36.4 million litas, the word that carries the sentiment is boosted, and no interpretation of the digits is required. Sentiment analysis generally seeks to understand the context in which numbers are used rather than to extract and compute them. Dates and years, also common, are equally unnecessary. One thing was preserved, however: before the digits were deleted, the abbreviations denoting orders of magnitude, m, mln and mn for million and their equivalents for billion and trillion, were expanded into the full word. Keeping those magnitude words retains the original meaning and can help predict sentiment.

Whitespace was removed last, both the whitespace present in the raw files and the extra whitespace created by the operations above.

Text wrangling

Normalisation then applied three of the four standard operations, and the one that was skipped is the interesting one.

Lowercasing was applied, consolidating duplicates such as THE, The and the.
Stop words were not removed. Some of them, not, more, very and few among them, carry meaning in financial text that is directly useful for sentiment. Articles could safely have gone, but to avoid confusion nothing was removed at this stage. The question was deferred to the exploration stage, where frequency analysis would identify custom stop words specific to this corpus.
Stemming was applied, being simple to perform and appropriate for a sentiment model.

Whitespace was stripped again afterwards. One further normalisation was applied on domain grounds: different currency abbreviations such as EUR and USD were folded into a single token, currencysign, and the dollarsign annotation substituted earlier during cleansing was folded into the same token. That removes tokens which are distinct but redundant while keeping their meaning intact, and it is the kind of substitution that only domain knowledge suggests.

One sentence through the full text pipeline

Stage	Text	Sentiment
Raw	Profit before taxes was EUR 4.0 mn , down from EUR 4.9 mn .	negative
Cleansed	Profit before taxes was EUR million down from EUR million	negative
Normalised	profit befor tax was currencysign million down from currencysign million	negative

Stemming has turned before into befor and taxes into tax; the two currency codes have collapsed into one token; the magnitude word million has survived by design.

Tokenising the normalised text produced 2,673 unique tokens. Those tokens together are the preliminary unigram bag-of-words for the corpus, and a document term matrix could be built from them straight away. It would be premature, though. Exploration may reveal uninformative tokens and anomalies, and any token that has to be removed also changes which n-grams can be formed. The final DTM therefore waits until after exploratory analysis and feature selection.

Exploratory data analysis on the corpus

Word counts across all sentences are computed first, and they are used to find outlier tokens at both ends: the words most commonly present and the words least commonly present. Ranking the corpus by occurrence puts the, currencysign, million, in, to, of, from, and, a, profit, for, it, net, oper, year, sale and percentsign at the top. These are the words that will be removed at the feature selection step. Two of them are worth noticing: million and currencysign rank that high only because the corpus is financial, which is a reminder that a custom stop word list is corpus-specific and cannot be borrowed.

Splitting the same count by sentiment class is more revealing still. The most frequent words in the negative-sentiment sentences and in the positive-sentiment sentences turn out to be very nearly the same list. A token that appears at the top of both lists cannot discriminate between them, which is a direct demonstration of why the most common tokens have to go.

A histogram of sentence length, measured as the number of characters including spaces, shows the longest sentence at 273 characters, the shortest at 26, and an average of about 120 characters. This distribution has no direct effect on training. Its role is diagnostic: it shows the spread and would flag any extremely long or short sentence for removal. Nothing here looks unusual, so no sentences were dropped.

Word clouds finish the exploration, because they let a large number of tokens and their weights be taken in at a glance. A cloud of the whole corpus sizes each word in proportion to its number of occurrences. Splitting the cloud into a negative half and a positive half is where the value appears: highly discriminative stems show up immediately, the negative half foregrounding decreas alongside down while the positive half foregrounds increas alongside rose. Those are exactly the tokens feature selection should keep, and the common words are exactly the ones it should discard.

Feature selection through frequency analysis

Two groups of tokens are candidates for removal. The frequent ones have already been identified. The other group is the rare tokens, many of them proper nouns, which say nothing about the sentiment of a sentence. Across the 2,180 sentences there are 44,151 non-unique tokens in total.

Frequency analysis quantifies how important each token is, both within a sentence and across the corpus. Term frequency at the corpus level, also called collection frequency, is the number of times a word appears in the whole corpus divided by the total number of words in it. Because it is a ratio of occurrences to total words, it necessarily lies between 0 and 1. The descriptive statistics of collection-level TF for this corpus are as follows.

Summary statistics of term frequency at the collection level

Statistic	Value
Minimum	2.265e-05
First quartile	2.265e-05
Median	4.530e-05
Mean	3.741e-04
Third quartile	1.585e-04
Maximum	5.429e-02

The mean sits well above the third quartile, which is the numerical signature of a long right tail.

The six highest collection-level TF values in the corpus belong to the, at 0.05429096, million at 0.04722430, currencysign at 0.04627302, in at 0.03870807, to at 0.03476705 and of at 0.03377047. At the other extreme sit tokens such as yet, yihn, young, zahariev, zone and zoo, every one of them at 2.264954e-05, which is the value produced by a single occurrence in 44,151 tokens.

A histogram of TF values makes the shape of the problem visible. There is a long tail to the right, made up of the common terms that must go, and there are very tall bars on the left, made up of terms that appear only once or twice across the whole dataset. Those rare terms do not appear often enough to work as features and are usually removed as well.

Figure 6: Both tails of the term frequency distribution are noise

Selection cuts from both ends. The left cut removes tokens too rare to generalise, the right cut removes tokens too common to discriminate, and only the middle band carries usable signal.

After examining the words at both extremes, two cuts were made before the final DTM was built: the high-TF words above the 99.5th percentile, 14 words in all, and the low-TF words below the 30th percentile, 714 words. The 14 high-TF words are the custom stop words for this project: the, million, currencysign, in, to, of, from, a, and, profit, for, it, not and year. Single-character letters were also eliminated, since a lone letter adds nothing to sentiment.

The frequency measures table

Populating a DTM requires more than a raw count, so a term frequency measures table is computed for every token in every sentence. Its columns are as follows.

SentenceNo. A unique identifier for each sentence, numbered in the order the sentences appear in the original dataset.
TotalWordsInSentence. The count of words in that sentence. Sentence 701, for example, holds 39 words.
Word. The token present in that sentence.
TotalWordCount. The number of occurrences of that token in the entire collection. The token the occurs 2,397 times across all sentences.
WordCountInSentence. The number of times the token appears in that particular sentence. The token the appears six times in sentence 701.
SentenceCountWithWord. The number of sentences containing the token. The token the appears in 1,453 sentences.

Four ratios are then derived from those counts.

Document frequency. Words that occur across many sentences differentiate none of them.

Inverse document frequency, and the product that combines sentence-level TF with corpus-wide IDF.

The logic of the last measure repays attention. IDF is a relative measure of how unique a term is across the whole corpus, and its meaning does not depend directly on the size of that corpus. A low IDF signals a word that appears everywhere. Multiplying sentence-level TF by IDF gives a complete picture of the value of a word: a high TF–IDF marks a word that occurs often inside a small number of documents, which is to say a relatively unique and important term, while a low TF–IDF marks a word spread across many documents. TF–IDF values can therefore serve directly as feature values for training.

Frequency measures for high-frequency tokens

Sentence	Words in sentence	Word	Total word count	Count in sentence	Sentences with word	TF	DF	IDF	TF–IDF
624	34	a	873	6	687	0.1764706	0.3151376	1.1547459	0.20377868
701	39	the	2397	6	1453	0.1538462	0.6665138	0.4056945	0.06241454
1826	34	a	873	6	687	0.1764706	0.3151376	1.1547459	0.20377868
1963	39	the	2397	6	1453	0.1538462	0.6665138	0.4056945	0.06241454
128	30	of	1491	5	984	0.1666667	0.4513761	0.7954543	0.13257571
223	37	the	2397	5	1453	0.1351351	0.6665138	0.4056945	0.05482358

Frequency measures for high TF–IDF tokens

Sentence	Words in sentence	Word	Total word count	Count in sentence	Sentences with word	TF	DF	IDF	TF–IDF
28	7	risen	3	1	3	0.1428571	0.0013761468	6.588468	0.9412097
830	7	diminish	2	1	2	0.1428571	0.0009174312	6.993933	0.9991333
1368	9	great	4	1	4	0.1111111	0.0018348624	6.300786	0.7000873
1848	8	injuri	1	1	1	0.1250000	0.0004587156	7.687080	0.9608850
1912	7	cheaper	1	1	1	0.1428571	0.0004587156	7.687080	1.0981543
1952	6	argument	1	1	1	0.1666667	0.0004587156	7.687080	1.2811800

Compare the two tables. The first holds tokens with high TF and low IDF; the second holds tokens with modest TF but very high IDF, and it is the second group that carries meaning.

For this project the cells of the DTM were filled with TF values rather than TF–IDF. The reason is that the documents here are single sentences rather than paragraphs or longer bodies. TF–IDF varies with the number of documents in the dataset, so a model trained on TF–IDF could behave differently when applied to a collection containing only a few documents.

Feature engineering on the corpus

N-grams supply the engineering step. Unigrams and bigrams were both implemented and the BOW built from the two together. The point of the bigrams is to hold negations intact, which is essential for sentiment: not and good become not_good, and no and longer become no_longer. In each case a sequence that a unigram BOW would destroy survives as a single token. The final BOW contains tokens such as increas, loss, loss_prior, oper_rose, tax_loss and sale_increas, all of which say something about embedded sentiment, and the frequency measures for the DTM were recomputed on this new BOW.

Example 8 · Worked

Two data scientists, Jack and Jill, are building sentiment indicators from financial text and have pulled the frequency measures tables shown above. Work with the token a in sentence 1,826 and the token great in sentence 1,368. The corpus contains 44,151 words in 2,180 sentences.

1. Determine and interpret term frequency at the collection level for both tokens.

Solution. Collection-level TF is the total count of the word divided by the total words in the collection.
For a: 873 ÷ 44,151 = 0.019773, or 1.977%.
For great: 4 ÷ 44,151 = 0.000091, or 0.009%.

The measure states, as a percentage, how much of the whole collection of 44,151 words a given token accounts for. Its use is to identify outlier words. The highest values belong mostly to stop words such as a, which are present in nearly every sentence and therefore cannot differentiate sentiment. The lowest values belong mostly to proper nouns and sparse terms, which are not important to the meaning either. It is the tokens in between that potentially carry information about sentiment.

2. Determine and interpret term frequency at the sentence level for both tokens.

Solution. Sentence-level TF is the count of the word in that sentence divided by the total words in that sentence.
For a in sentence 1,826: 6 ÷ 34 = 0.176471, or 17.647%.
For great in sentence 1,368: 1 ÷ 9 = 0.111111, or 11.111%.

This measure states how much of one particular sentence a token accounts for, so it speaks to the importance of that token within that instance rather than across the corpus. Note that on this measure a and great look broadly comparable, which is exactly why the sentence-level figure cannot be used on its own.

3. Determine and interpret document frequency and inverse document frequency for both tokens.

Solution. DF is the number of sentences containing the word divided by the 2,180 sentences in the corpus.
For a: 687 ÷ 2,180 = 0.315138, or 31.514%.
For great: 4 ÷ 2,180 = 0.001835, or 0.184%.

IDF is the logarithm of the reciprocal of DF.
For a: log(1 ÷ 0.315138) = 1.154746.
For great: log(1 ÷ 0.001835) = 6.300786.

DF matters because a token spread across a third of all sentences, as a is, provides no differentiating information in any one of them. A token confined to four sentences out of 2,180, as great is, may provide a great deal. IDF simply inverts and compresses that reading, so the uninformative token scores low and the distinctive token scores high.

4. Determine and interpret TF–IDF for both tokens.

Solution. TF–IDF multiplies sentence-level TF by IDF.
For a in sentence 1,826: 0.176471 × 1.154746 = 0.203779, or 20.378%.
For great in sentence 1,368: 0.111111 × 6.300786 = 0.700087, or 70.009%.

Here is the payoff. On sentence-level TF alone, a scored higher than great, at 17.647% against 11.111%, which would have been exactly the wrong ranking. Multiplying by IDF reverses it decisively, 70.009% against 20.378%, because great appears many times within a very small number of documents while a appears in most sentences and discriminates nothing. TF–IDF is what extracts the key terms of a document for use as training features.

5. Verify the collection-level TF of the token the against the summary statistics reported for the corpus.

Solution. The token the occurs 2,397 times in a collection of 44,151 words:
2,397 ÷ 44,151 = 0.05429096.
That is the maximum value in the summary statistics table, reported there as 5.429e-02. The most frequent token in the corpus is the, which is what the custom stop word list reflects. Its DF confirms the same thing from the other direction: 1,453 ÷ 2,180 = 0.6665138, so 66.7% of all sentences contain it.

The sentiment labels are the target variable. They are recoded as 1 for positive and 0 for negative so that ROC curves and AUC can be computed from the model output. The cleansed and preprocessed master dataset is then partitioned into a training set, a cross-validation set and a test set in the ratio 60:20:20, following common practice. Splitting uses simple random sampling within the levels of the target variable, which keeps the class distribution roughly the same inside each split rather than leaving it to chance.

Building the three document term matrices

The final DTM for training is built with the training sentences as rows, meaning the instances, and the tokens from the training BOW as columns, meaning the feature variables. That BOW consists of unigram and bigram tokens taken from the training corpus only, and the matrix is filled with TF values computed on the training corpus.

The DTMs for the cross-validation set and the test set are then built using the tokens from that same final training BOW. The columns of all three matrices are therefore identical, while the row counts differ because each split holds a different number of sentences. Each matrix is filled with term frequency values calculated from the sentences in its own corpus.

Why the validation and test matrices are built from the training vocabulary

The trained model is nothing more than a set of coefficients, one per token in the training BOW. A coefficient exists only for a token the model has seen. If the validation matrix were built from its own vocabulary it would contain columns the model has no coefficient for, and would be missing columns the model expects, so the two objects would not multiply together. Reusing the training vocabulary also protects the evaluation itself: allowing the validation or test set to contribute vocabulary would let information from data the model is supposed to be judged on leak into the model definition, and the resulting performance estimate would flatter the model. The row counts differ freely, because rows are observations and adding an observation changes nothing about the model.

Dimensions of the three splits

Corpus	Split	Sentences	DTM dimensions	Purpose
Master	100%	2180	2180 × 11501	Used for data exploration
Training	60%	1309	1309 × 9188	Used to train the model
Cross-validation	20%	435	435 × 9188	Used to tune and validate the trained model
Test	20%	436	436 × 9188	Used to test the trained, tuned and validated model

The column count of 9,188 is the number of unique tokens in the final training BOW. It differs from the 11,501 in the master corpus because random sampling put only some sentences into the training split, and the tokens unique to the other sentences never entered the training vocabulary.

Method selection

Support vector machines, decision trees and logistic regression were all examined, since all three suit this combination of supervised learning, text data and wide data with many potential variables. The support vector machine and the logistic regression outperformed the decision trees, and logistic regression is the one carried forward.

Text data typically contain thousands of tokens, which makes a DTM sparse: each column is one token feature and most of the cells are zero, because most tokens do not appear in most texts. Logistic regression copes with sparse training data because the coefficient on a token that appears in few sentences will be close to zero, so the model effectively ignores a very large number of barely useful features. Regularisation pushes the same coefficients down further.

Because the method uses maximum likelihood estimation, its output is a probability between 0 and 1. The target variable is binary, so the coefficients are not used to predict the target value directly. Instead a mathematical function of the coefficients returns the probability p that a sentence carries positive sentiment. A sentence with p of 0.90 has a 90% likelihood of being positive. In theory sentences with p above 0.50 are the positive ones, but that is not reliably true in practice, so the threshold value of p is itself something to be found. The ideal threshold depends on the dataset and on how the model was trained, and it is estimated heuristically from performance metrics and ROC curves.

Evaluation and regularisation

Running the trained model over the training and cross-validation matrices produces two ROC curves that look distinctly different from one another. The AUC is 96.5% on the training data and 86.2% on the cross-validation data. The model clearly handles the data it was fitted to, so underfitting is not the issue. The gap between the two figures is the issue: a materially higher error rate out of sample than in sample is the definition of overfitting.

The remedy applied is LASSO regularisation, standing for least absolute shrinkage and selection operator, applied to the logistic regression. LASSO penalises the coefficients so as to prevent overfitting. In doing so it performs selection as well as shrinkage: it keeps the tokens whose coefficients are statistically significant, meaning non-zero, and which contribute to the fit, and disregards the rest.

The regularised model produces ROC curves that look alike on both datasets, with an AUC of 95.7% on training data and 94.8% on cross-validation data. Similar performance on both indicates a well-fitted model. The training AUC has fallen slightly, from 96.5% to 95.7%, and that is not a cost worth worrying about: the cross-validation AUC, which is the figure that predicts behaviour on new data, has risen by more than eight percentage points.

A third model was also trained, using every token feature including stop words, sparse terms and single characters, with no regularisation at all. It returned an AUC of 99.1% on the training data and 89.4% on the cross-validation data, which is the overfitting signature again and in a more extreme form. Since the AUC values across all three models are not far from 100%, none of them is underfitting. The comparison isolates the point neatly: regularisation together with careful feature selection is what prevents overfitting in a logistic regression, and the final model for this project is logistic regression with LASSO regularisation.

Tuning the threshold

Error analysis on the cross-validation results begins at the default cutoff of 0.50, so that a sentence with p above 0.50 is predicted positive and any other sentence is predicted negative.

Example 9 · Worked

The cross-validation set contains 435 sentences. At a threshold of 0.50 the model produces 284 true positives, 38 false positives, 7 false negatives and 106 true negatives. At a threshold of 0.60 it produces 284 true positives, 30 false positives, 7 false negatives and 114 true negatives.

1. Compute the performance metrics at a threshold of 0.50.

Solution.
Precision = 284 ÷ (284 + 38) = 284 ÷ 322 = 0.88.
Recall = 284 ÷ (284 + 7) = 284 ÷ 291 = 0.98.
F1 score = (2 × 0.88 × 0.98) ÷ (0.88 + 0.98) = 1.7248 ÷ 1.86 = 0.93.
Accuracy = (284 + 106) ÷ (284 + 38 + 106 + 7) = 390 ÷ 435 = 0.90.
Model accuracy is therefore 90% at the theoretically suggested default threshold.

2. Compute the same metrics at a threshold of 0.60 and state what has changed.

Solution.
Precision = 284 ÷ (284 + 30) = 284 ÷ 314 = 0.90.
Recall = 284 ÷ (284 + 7) = 284 ÷ 291 = 0.98.
F1 score = (2 × 0.90 × 0.98) ÷ (0.90 + 0.98) = 1.764 ÷ 1.88 = 0.94.
Accuracy = (284 + 114) ÷ (284 + 30 + 114 + 7) = 398 ÷ 435 = 0.91.

Raising the cutoff has moved eight sentences out of the false positive cell and into the true negative cell, while leaving the true positive and false negative counts untouched. Accuracy and F1 have each gained one percentage point, to 91% and 94%, and precision has gained two, to 90%. Recall is unchanged at 98%, because neither of the quantities in its formula moved.

3. How was the value of 0.60 arrived at?

Solution. By systematic search on the cross-validation data. Threshold values from 0.01 to 0.99 were evaluated one at a time, with a confusion matrix and a full set of performance metrics computed at each. The value producing the highest accuracy was then selected. Plotting F1 and accuracy against the threshold puts the peak at around 0.60, and a detailed table across the range 0.45 to 0.75 shows the highest accuracy and F1 arising for thresholds between 0.60 and 0.63, from which 0.60 was chosen.

Two things justify optimising on accuracy and F1 here. Both give equal weight to false positives and false negatives, and in this project the values of a negative-sentiment and a positive-sentiment sentence are assumed equal, so the consequences of the two error types are equal too. Where that assumption does not hold, the priority between precision and recall has to be set from the application, not from the data.

4. The model is now applied to the test set of 436 sentences, giving 284 true positives, 35 false positives, 7 false negatives and 110 true negatives. Compute the metrics and comment.

Solution.
Precision = 284 ÷ (284 + 35) = 284 ÷ 319 = 0.89.
Recall = 284 ÷ (284 + 7) = 0.98.
F1 score = (2 × 0.89 × 0.98) ÷ (0.89 + 0.98) = 1.7444 ÷ 1.87 = 0.93.
Accuracy = (284 + 110) ÷ 436 = 394 ÷ 436 = 0.90.

The test figures sit alongside the cross-validation figures with barely any deterioration, and the ROC evidence already showed training and cross-validation performance to be similar. Consistency across all three splits is the evidence that the model is robust, is not overfitting, and should generalise to new sentences from comparable financial sources.

Performance measures across a range of threshold values, cross-validation data

Threshold	Precision	Recall	F1	Accuracy
0.45	0.8750000	0.986254296	0.927302100	0.8965517
0.50	0.8819876	0.975945017	0.926590538	0.8965517
0.55	0.8930818	0.975945017	0.932676519	0.9057471
0.59	0.9015873	0.975945017	0.937293729	0.9126437
0.60	0.9044586	0.975945017	0.938842975	0.9149425
0.63	0.9041534	0.972508591	0.937086093	0.9126437
0.70	0.9061489	0.962199313	0.933333333	0.9080460
0.75	0.9072848	0.941580756	0.924114671	0.8965517

Selected rows from the full table, which runs in steps of 0.01 from 0.45 to 0.75. Precision climbs steadily with the threshold while recall falls, and accuracy and F1 both peak in the band from 0.60 to 0.63.

Applying the model and reading the output

The validated model is now ready. It is nothing more than a collection of penalised regression coefficients for the unigram and bigram tokens of the training BOW. To use it on new sentences, those sentences must be tokenised, cleansed and preprocessed exactly as the training data were: every operation performed on the training corpus has to be repeated identically on anything the model is applied to. The model then applies its coefficients to the term frequency values in the new DTM and returns a target value p, and the threshold of 0.60 converts p into a predicted class.

Applied to the 436 test sentences, which were preprocessed alongside the rest of the master corpus but never used for training or tuning, the model produces a p value and a predicted sentiment for each. A sample of the output shows what success and failure look like in practice.

Sample results from the test corpus

Processed sentence	Actual	p	Predicted
refin margin was bbl combar bbl prior	1	0.81	1
deal is like bring save	1	0.83	1
pre tax loss total compar loss first quarter	1	0.46	0
ebit total compar loss correspond period	1	0.74	1
compani expect sale signific increas	1	0.91	1
third quarter fiscal efor swung loss versus correspond period fiscal	0	0.77	1
loss after financi item total compar correspond period	0	0.71	1
last paseng flew airlin down percent	0	0.12	0

Rows selected from the 30-sentence sample published for the project. In that full sample there are three misclassifications, at p values of 0.46, 0.77 and 0.71, so the accuracy of the 30-text sample is 27 ÷ 30 = 90%.

The misclassified rows are instructive. Each of them contains the token loss, which the model has learned to associate with negative sentiment, yet the surrounding words reverse the reading in ways a unigram and bigram model struggles to catch. A sentence describing a result that improved relative to a prior loss is positive; a sentence describing a swing into loss is negative; the vocabulary is nearly identical in both.

What the project delivers

Stepping back, the project converted unstructured data, financial text, into structured data, meaning tokens, sentences and term frequency values organised into a document term matrix, and used that matrix to train, validate and test a machine learning model, here a logistic regression, for a classification task, here sentiment class. The same architecture transfers to other contexts where sentiment has to be extracted from longer texts.

The resulting classification has two uses. It works as a visualisation aid, giving insight into the tone of a body of documents without anyone having to read them. It also works as structured input to a larger model with a specific objective, such as predicting future share price movements, which is where the sentiment score stops being an end in itself and becomes a feature.

Example 10 · Worked

Two further confusion matrices are constructed on the cross-validation data, one at a threshold of 0.75 and one at 0.45, and are to be compared with the selected model at 0.60.

Three confusion matrices compared

Metric	Matrix A, threshold 0.75	Matrix B, threshold 0.45	Selected model, threshold 0.60
True positives	281	281	284
False positives	28	41	30
False negatives	17	4	7
True negatives	110	110	114
Precision	0.91	0.87	0.90
Recall	0.94	0.99	0.98
F1 score	0.93	0.93	0.94
Accuracy	0.90	0.90	0.91

1. Compare matrix A with the selected model.

Solution. Matrix A has fewer true positives and fewer true negatives than the selected model, 281 against 284 and 110 against 114, so both of its overall measures come out lower: accuracy 0.90 against 0.91 and F1 0.93 against 0.94. Its precision is marginally better, 0.91 against 0.90, because it produces slightly fewer false positives. That gain is bought at a heavy price in recall, 0.94 against 0.98, because false negatives rise from 7 to 17. On balance the model at a threshold of 0.60 is superior.

2. Compare matrix B with the selected model.

Solution. Matrix B has the same true positive and true negative counts as matrix A, 281 and 110, so it too returns lower accuracy at 0.90 and a lower F1 at 0.93. Its recall is slightly better, 0.99 against 0.98, because it produces fewer false negatives. Its precision is worse, 0.87 against 0.90, because false positives rise from 30 to 41. Again the threshold of 0.60 gives the better model.

3. Contrast A with B and explain the trade-off they illustrate.

Solution. Accuracy and F1 are identical across the two, at 0.90 and 0.93. The whole difference sits in precision and recall: A has the higher precision at 0.91 against 0.87, B has the higher recall at 0.99 against 0.94. That is the false positive against false negative trade-off in its clearest form, and moving the threshold is what walks along it.

Which end to favour depends entirely on the cost of each error. Precision matters where a false positive is expensive, as when a sound and expensive product fails inspection and is scrapped. Recall matters where a false negative is expensive, as when a defective product passes inspection and reaches the customer. Translated into sentiment classification: a false positive might lead an investor to buy a stock whose sentiment was actually negative, while a false negative might lead the investor to avoid or even short a stock whose sentiment was actually positive. The model behind the selected 0.60 threshold strikes the balance between the two.

Check yourself

1. A researcher worries that a large share of the social media posts feeding a model come from fake accounts. Which characteristic of big data does that concern relate to, and why is it the one that big data amplifies?

Veracity, the credibility and reliability of a source. Volume, variety and velocity describe how much data there is, how many forms it takes and how fast it arrives; none of them speaks to whether it can be trusted. Veracity becomes critical for big data precisely because the datasets are assembled from so many different sources, and the estimates cited in the curriculum are that 10% to 15% of social content is entirely fake while web spam is reckoned to make up over 20% of everything on the worldwide web.

2. A raw table lists one borrower with an income of −63,050 and another whose loan amount reads unknown even though the loan is recorded as repaid. Classify each defect and explain the difference.

The negative income is an invalidity error, because the value lies outside any meaningful range. The unknown loan amount is an inaccuracy error, because the entry does not measure the true value even though nothing about it is out of range or missing. The distinction is that invalidity can be spotted from the value alone, while inaccuracy can only be spotted by knowing that the lender must in fact know how much it advanced.

3. A modeller creates a feature named income_above_100k taking the values 0 and 1, and separately drops the Name column because ID already identifies each row. Name both operations correctly.

Creating income_above_100k is feature engineering, specifically the decomposition of a continuous variable into a band using domain knowledge about tax brackets. Dropping the Name column is selection, one of the five preprocessing transformations, and it belongs to the preparation stage rather than the exploration stage. Feature selection is a different activity again: it is statistical, uses measures such as chi-square and information gain, and follows exploratory data analysis.

4. Why can removing stop words fail to remove them once n-grams are in use, and when is that failure actually useful?

A stop word list contains isolated words, so it cannot match a token such as to_the in which two stop words have been welded into a single bigram. The failure is useful in sentiment work, where negations must survive. Removing no as an isolated stop word destroys the meaning of a phrase such as no market for the product, whereas the bigrams not_good and no_longer keep the negation attached to the word it negates.

5. A token has a sentence-level term frequency of 0.111111 and an inverse document frequency of 6.300786, while a second token in another sentence has a sentence-level term frequency of 0.176471 and an inverse document frequency of 1.154746. Which is the more useful feature, and why would sentence-level TF alone have misled you?

The first token is far more useful. Its TF–IDF is 0.111111 × 6.300786 = 0.700087, against 0.176471 × 1.154746 = 0.203779 for the second. Sentence-level TF alone ranks the second token higher, at 17.647% against 11.111%, because it counts occurrences within one sentence and says nothing about the corpus. IDF supplies the missing information: a high IDF means the token is confined to very few documents and is therefore distinctive, while a low IDF means it is spread everywhere and discriminates nothing.

6. A model returns an AUC of 99.1% on the training data and 89.4% on the cross-validation data. Diagnose the problem, state which type of error is high, and name two remedies.

The model is overfitted: performance is far better in sample than out of sample. Variance error is high and bias error is low, because the model has memorised the training data rather than learned the pattern. Two remedies are regularisation, such as LASSO applied to a logistic regression, which penalises the coefficients and drops the tokens that do not contribute, and tighter feature selection, using measures such as chi-square and mutual information to cut the feature count. Adding training instances would also help. Underfitting is not the issue, since an AUC near 100% in sample rules it out.

7. Two thresholds produce identical accuracy of 0.90 and identical F1 of 0.93, but one gives precision of 0.91 and recall of 0.94 while the other gives precision of 0.87 and recall of 0.99. How would you choose between them?

Not from the statistics, because the overall measures are tied. The choice is a business judgement about the relative cost of the two error types. Favour the higher-precision model where a false positive is expensive, for instance where acting on a wrongly positive sentiment signal means buying a stock that should have been avoided. Favour the higher-recall model where a false negative is expensive, for instance where a wrongly negative signal means avoiding or shorting a stock whose sentiment was in fact positive. Where the two costs are genuinely equal, as this project assumes, a third threshold that lifts both overall measures is preferable to either.

8. The master corpus yields 11,501 unique tokens but the training document term matrix has only 9,188 columns, and the cross-validation and test matrices have 9,188 columns too. Explain both facts.

The training BOW is built from the training sentences alone, and random sampling placed only 1,309 of the 2,180 sentences into that split, so tokens appearing exclusively in the other 871 sentences never entered the training vocabulary. The validation and test matrices then reuse that same 9,188-token vocabulary because the model consists of one coefficient per training token: a column the model has no coefficient for cannot be scored, and letting the held-out data contribute vocabulary would leak information into the model and inflate its measured performance. Only the row counts differ, at 435 and 436, because rows are observations.

End of lesson.

Back to 01 Quantitative Methods

QM 7 – Big Data Projects

Overview

Big data and the four Vs in investment management

What makes data big

Why this matters for fintech

The steps in a data analysis project

The traditional pipeline, starting from structured data

The text pipeline, starting from unstructured data

Preparing and wrangling structured data

Before collection begins

The six error types

Five preprocessing transformations

Outliers

Scaling

Preparing and wrangling unstructured text data

Text cleansing

Tokens, tokenisation and normalisation

Bag-of-words and the document term matrix

N-grams

Exploring structured data

Exploratory data analysis

Feature selection

Feature engineering

Exploring text: feature selection and feature engineering

Exploratory analysis of text

Selecting features from text

Engineering features from text

Model training: fit, method selection and data splits

Structured and unstructured data train the same way

Evaluating the fit of an algorithm

What causes a bad fit

Method selection

Splitting the data

Class imbalance

Performance evaluation

Error analysis and the confusion matrix

Receiver operating characteristic

Root mean squared error

Tuning and the bias-variance trade-off

Parameters and hyperparameters

Fitting curves

When tuning is not enough

A financial forecasting project: curation, wrangling and exploration

Text curation

Text preparation

Text wrangling

Exploratory data analysis on the corpus

Feature selection through frequency analysis

The frequency measures table

Feature engineering on the corpus

A financial forecasting project: training, tuning and results

Building the three document term matrices

Method selection

Evaluation and regularisation

Tuning the threshold

Applying the model and reading the output

What the project delivers