PORT 6 – Backtesting and Simulation

Portfolio managers constantly propose new ideas: a valuation screen, a momentum overlay, a factor combination built for diversification. Before any of them reach client capital, the manager needs evidence that the idea has merit. This reading develops four complementary techniques for building that evidence. Backtesting asks how a strategy would have performed had it been run over a historical period. Historical scenario analysis narrows that question to specific historical episodes, such as a recession or a stretch of unusually high volatility. Simulation steps outside history altogether and explores how a strategy would behave in a hypothetical environment chosen by the analyst, which matters because history is only one realised path out of many that could have occurred. Sensitivity analysis, usually paired with simulation, tests how much the conclusions change when a key modelling assumption is varied.

These techniques have become far more accessible than they once were. A CFA Institute survey of close to 250 analysts, portfolio managers and private wealth managers found that half had run a backtest on an investment strategy within the previous twelve months. Wider access to the software has not reduced the importance of understanding what these tools assume, where they mislead, and how to read their output. If anything the stakes are higher, because a manager can now produce an impressive-looking backtest without having thought through any of that.

Backtesting rests on an implicit assumption: that the future will resemble history at least somewhat. Because that assumption is imperfect, a strategy that performs poorly in a backtest could still deliver excess returns going forward, and a strategy that performs well in a backtest could still fail to do so later. In practice, however, few investment committees will approve a strategy that cannot clear a backtest, so backtesting functions as close to a screening hurdle regardless of how carefully the theoretical caveats are worded. It suits quantitative and systematic strategies most naturally, because their rules translate directly into a historical simulation, but fundamental managers use it too, commonly to check whether a valuation or quality criterion would have added value before building it into a stock screen.

Three steps, always in sequence

Every backtest proceeds through the same three stages, regardless of how simple or elaborate the underlying strategy is: strategy design, historical investment simulation, and analysis of the backtesting output.

Strategy design and its key parameters

Strategy design begins with an investment hypothesis: a trading rule, a security selection criterion, or a portfolio construction method believed to earn superior risk-adjusted absolute return or excess return against a benchmark. That hypothesis must then be translated into rules precise enough to run through history, which requires the analyst to fix several parameters.

The investment universe is the full set of securities eligible for the strategy. Academic researchers commonly combine the Compustat/Worldscope and CRSP databases, but practitioners more often use the constituents of well-known broad market indexes, such as the Russell 3000 for the United States, the S&P/TSX Composite for Canada, MSCI China A for mainland China, and the S&P Global Broad Market Index for other markets.

The return definition matters once the universe crosses borders. A global strategy has to settle on a currency convention: either translate every return into one reference currency, typically the manager’s home currency, or leave returns denominated locally. The choice usually tracks whether the manager hedges currency exposure; a manager who does not hedge tends to backtest in single-currency terms so that the reported results already embed the currency risk the strategy will actually carry.

The rebalancing frequency is most often monthly, though higher and lower frequencies both appear in practice. Rebalancing more often raises transaction costs, and daily or higher-frequency data introduces its own distortions: bid–offer spreads, trading that is not synchronised across time zones, and gaps from holidays that differ by country. Because many apparent market anomalies disappear once realistic transaction costs are applied, an analyst presenting backtest output should always state plainly whether costs are included.

The start and end date pulls in two directions. A longer history gives more statistical confidence, but financial data are not stationary, so a long-run average can obscure how a strategy behaved in specific regimes. The standard remedy is to supplement a long backtest with historical scenario analysis of shorter, discrete regimes, covered later in this reading.

Once the parameters are fixed, the second stage runs the strategy through history. Analysts typically use a rolling, or walk-forward, window: the portfolio is formed at each period end using only data available up to that point, held over a subsequent out-of-sample period, and then reformed as new information arrives. The process replicates how live investing actually works, because a manager continually updates positions as fresh data comes in rather than acting once on a fixed snapshot.

Constructing a long–short quantile portfolio

A common way to translate a single valuation signal into a testable portfolio is the long–short hedged approach popularised by Fama and French. The universe is sorted by the metric under study, in this case trailing earnings yield, and split into quantiles, most often quintiles or deciles. The portfolio goes long the quantile with the strongest scores and short the quantile with the weakest, with names inside each quantile either equally weighted or weighted by market capitalisation. The middle quantiles are simply left out. Because the top and bottom quantiles may differ in beta and in exposure to other common factors, the spread in average earnings yield between them is only a rough, though useful, proxy for the strategy’s true underlying exposure.

Reading the output: metrics and visuals

Once returns have been recorded period by period, the analysis stage looks at both the average return and the risk profile. Standard metrics include the Sharpe ratio, the Sortino ratio, volatility, and maximum drawdown, defined as the largest peak-to-trough loss the strategy suffers at any point in the sample. Visuals carry information a single summary statistic cannot: a time series of returns, a distribution of returns plotted against a reference distribution such as the normal, and, especially useful for a long backtest, cumulative performance on a logarithmic scale, where equal percentage moves occupy equal vertical distance. A log-scale cumulative chart makes it easy to spot deteriorating performance and structural breaks that a linear chart would compress or exaggerate depending on the starting level.

Structural breaks matter because they are one of the main reasons the past is an imperfect guide to the future. Recognisable sources include recessions and depressions, such as the 2008–09 global financial crisis; geopolitical shifts, including changing trade relationships and countries entering or leaving major trading blocs; major turns in monetary or fiscal policy, such as the extended period of quantitative easing that followed the 2008 crisis; and large technological shifts, from the dot-com era to the more recent spread of machine learning.

The shape of the US return distribution over this period shows excess kurtosis and negative skewness relative to the normal curve. Excess kurtosis means extreme outcomes, in either direction, are more common than a normal distribution implies; negative skewness means those extremes are more likely to fall on the downside. Together they explain why the maximum drawdown figure above runs well beyond what volatility alone would predict.

Few managers rely on a single signal. Most quantitative equity models, and most fundamental screening processes, combine several factors. This section introduces two ways of combining factor portfolios into a single multifactor strategy: an equally weighted benchmark (BM) portfolio and a risk parity (RP) portfolio that instead weights factors so that each contributes equally to total portfolio risk. Both weighting schemes are deliberately objective, which makes them useful teaching examples and a natural pair to compare.

Eight underlying factors

Each factor portfolio buys the top 20% of stocks and shorts the bottom 20% ranked by that factor, equally weighted within each side, ignoring transaction costs for illustration.

Over 1988–2019 in the United States, all eight factor portfolios delivered positive cumulative returns, but with wide dispersion between them. Earnings revision, earnings yield and price momentum produced the strongest results, while earnings growth and the debt-to-equity factor lagged well behind the rest. Visually the eight return series cluster into roughly three groups: earnings revision, earnings yield and price momentum tend to move together; return on equity and earnings quality form a second pair; and book-to-market, earnings growth and debt-to-equity form a third.

The BM portfolio combines the eight factor portfolios with equal weights. Research on portfolio construction has repeatedly found that naive equal weighting performs in line with, and sometimes better than, portfolios built from more elaborate optimisation techniques, which is one reason it is a sensible baseline. The RP portfolio instead weights each factor by its contribution to total portfolio risk, which requires estimating the volatility of every factor and the correlations between all of them. Both portfolios are long only: the weights assigned to each of the eight underlying long–short factor portfolios must be non-negative and sum to 100%, even though each factor portfolio inside them is itself long–short.

A second rolling window, layered on the first

Backtesting a multifactor strategy uses the same rolling-window logic as before, but applies it twice. The first layer forms the eight factor portfolios each month, exactly as described for a single-factor backtest. The second layer estimates the variance–covariance matrix that the RP weighting scheme needs, using only the trailing five years of monthly factor returns at each point in time. Using a trailing, rather than a full-sample, covariance estimate is what keeps the RP weights from incorporating information not yet available at the time each portfolio was actually formed.

Panel results over 1993–2019 in the United States show that RP weights are relatively stable through time but far from equal. Book-to-market and earnings quality typically receive the largest allocations; return on equity and price momentum typically receive the smallest. The RP portfolio’s cumulative return is lower than the BM portfolio’s, but its volatility is less than half of the BM portfolio’s, at 0.7% per month against 1.6% per month, so its Sharpe ratio comes out well ahead of the benchmark’s despite the lower raw return.

Backtest metrics are most informative side by side. The table below reports monthly return statistics for each of the eight factor portfolios and for the BM and RP multifactor portfolios, all measured over 1993–2019.

Six of the eight underlying factors carry negative skewness (earnings yield, earnings growth, momentum, earnings revision, ROE and debt-to-equity); only book-to-market and earnings quality are positively skewed. The BM portfolio inherits that negative skew at −2.40, while the RP portfolio is mildly positive at 0.51. Every column in the table shows kurtosis above 3.0, the normal distribution’s benchmark value, with earnings quality the least extreme at 3.87 and book-to-market the most extreme at 23.61. The BM and RP portfolios also stand out for how much smaller their standard deviations are than any individual factor’s, 1.6% and 0.7% per month respectively against a range of 1.2% to 4.6% among the eight underlying factors, which is the direct effect of diversifying across imperfectly correlated factor exposures.

Across all three downside measures, price momentum carries the worst downside risk of the eight factors, followed by return on equity: momentum posts the largest VaR (−8.4%), the largest CVaR (−22.9%) and the largest maximum drawdown (59.7%), with ROE second on each. At the other end, earnings quality has the smallest CVaR (−1.7%) and the smallest maximum drawdown (8.3%) of the eight factors, though book-to-market actually posts a marginally smaller VaR, −0.7% against earnings quality’s −1.3%. The two tail measures, CVaR and maximum drawdown, therefore favour earnings quality more clearly than VaR alone does. The RP portfolio has the smallest CVaR and the smallest maximum drawdown of every column in the table, and it ties book-to-market for the smallest VaR at −0.7%.

Three mistakes recur often enough in backtesting, and in quantitative research generally, that they deserve to be treated as a standard checklist: survivorship bias, look-ahead bias, and data snooping.

Survivorship bias

Index constituents change continuously. New companies enter through IPOs, spin-offs and strong performance; existing ones exit through privatisation, acquisition, bankruptcy, or a change in market capitalisation large enough to move them out of the index band. Fewer than 400 of the companies that made up the Russell 3000 in 1985, less than 13% of that original list, were still in the index as of 31 May 2019. The pattern is similar in Europe: the S&P BMI Europe Index started with about 720 names in 1989 and now holds around 1,200, but only 142 of the original 720, about 20%, were still constituents by May 2019.

Survivorship bias is the error of drawing conclusions from data that reflect only the entities that happened to survive to the observation date. It is one of the most obvious mistakes an analyst can make and, despite that, one of the most common, because it is straightforward, though not advisable, to backtest using only today’s index members. The problem is that nobody investing in the past could have known in advance which companies would survive, which would be delisted, and which had not even been created yet. The surviving list is also likely to be biased in identifiable ways, for example toward multinational, highly innovative, or otherwise unusually successful firms.

Point-in-time data, which tracks every company that was actually a constituent at each historical date, including the ones that later disappeared, is the correct remedy, and it is becoming more available from data vendors. The difference it makes is not academic: testing the low-volatility anomaly, the claim that low-volatility stocks outperform high-volatility stocks, with a proper point-in-time Russell 3000 universe confirms that low-volatility stocks outperformed over the three decades to 2019. Repeating the same test using only companies that survived to the end of the sample flips the conclusion entirely: high-volatility stocks then appear to outperform low-volatility stocks by roughly 5.5 times.

Look-ahead bias

Look-ahead bias is broader than survivorship bias, of which survivorship bias is actually one special case: it is the error of using information in a backtest that was not actually available at the historical date being tested. It takes three common forms. Reporting lags are the first: year-end 2018 earnings for the full universe would not all be available until roughly 31 March 2019, even though larger companies typically report earlier, so an analyst who assumes full availability at year end introduces look-ahead bias, while one who imposes too long a uniform lag introduces stale data for the companies that actually reported promptly. Data revisions are the second form: many databases overwrite historical macroeconomic and company figures with later restatements, so a backtest run today may unintentionally use information that was not available at the time it claims to represent. New database additions are the third form: when a vendor adds a company, it frequently backfills several years of financial history, so a backtest run on the current database can include companies that were not actually present in it during the period being tested. All three forms tend to push backtested results toward looking better than a genuinely point-in-time study would show.

The consequence is measurable. Testing the earnings yield strategy with reporting lag assumptions ranging from zero (full look-ahead bias) to six months against a true point-in-time benchmark shows that the zero-lag, look-ahead-biased version inflates US performance by almost 100%. The lag needed to match the point-in-time result also varies by region: one to two months in the United States, Canada and Japan; two to three months in Europe, the United Kingdom and Australia/New Zealand; and three months in Asia ex-Japan, Latin America and emerging EMEA, reflecting how quickly companies in each region typically report.

Data snooping

Data snooping, also called p-hacking, is making an inference after looking at the results rather than testing a hypothesis formed in advance. It shows up as backtesting many variants of a strategy and reporting only the one that happened to perform best, or repeatedly adjusting a model, dropping outliers, or adding variables until a result clears a significance threshold. The result is a strategy that looks statistically compelling in-sample purely because enough variants were tried, and is likely to disappoint out of sample.

Two mitigations help. Setting a higher significance hurdle than usual, for example requiring a t-statistic above 3.0 before accepting a newly discovered factor, raises the bar enough to filter out results that only look significant because many alternatives were tested. Cross-validation is the second: partition the data into a training set and a separate validation set, and only trust a model that also performs on the data it was not built on. Rolling-window backtesting is itself a form of cross-validation, since each in-sample period trains the model that is then tested on the following out-of-sample period, though it is deterministic rather than random. Testing a strategy across independent geographic markets is a practical version of the same idea: comparing the BM and RP portfolios across ten global markets over 1993–2019 shows RP delivering lower realised volatility than BM in every one of the ten markets, and a higher Sharpe ratio in seven of the ten, evidence that the RP result is not an artefact of the US sample alone.

A full-sample average can hide a great deal, because it blends periods that behaved very differently from one another. Historical scenario analysis, sometimes called historical stress testing, addresses this directly by measuring a strategy’s performance and risk inside specific historical regimes rather than across the whole sample at once. Two regime definitions recur most often: economic expansions against recessions, and high-volatility against low-volatility environments.

For the US strategies in this reading, the recession dating comes from the National Bureau of Economic Research, which identified two recessions over the RP strategy’s 1993–2019 sample: March 2001 to November 2001, and December 2007 to June 2009. An important caveat applies to using these dates directly in a backtest: they are identified only with hindsight, and often with a long lag. NBER did not call December 2007 the start of a recession until December 2008, a full year later, and did not confirm June 2009 as the end of that recession until September 2010, more than a year afterward. A backtest that assumes a manager could act on these dates in real time therefore carries its own look-ahead bias, even though it is a useful simplification for illustrating scenario analysis.

The volatility regime definition uses the CBOE’s VIX index, which measures options-implied volatility on the S&P 500. A rolling five-year moving average of the VIX is computed, and periods when the VIX sits above that moving average are labelled high-volatility, with periods below it labelled low-volatility.

How the BM and RP strategies respond to regime

Measured by Sharpe ratio, the RP strategy is notably robust across the recession and non-recession regimes, while the BM strategy’s Sharpe ratio deteriorates sharply during recessions. On the volatility split, BM performs slightly worse in low-volatility regimes than in high-volatility ones, while RP performs about equally well in both.

The shape of the return distributions tells a complementary story. In both strategies, the distribution of monthly returns is flatter, meaning higher standard deviation, during non-recession periods than during recessions. The BM strategy shows negative skewness and excess kurtosis, meaning a fat left tail, in both regimes, but its average monthly return is clearly lower during recessions than outside them. The RP strategy also earns a lower average return in recessions than outside them, but its volatility and kurtosis stay much lower than BM’s in both regimes, consistent with the diversification benefit already visible in the full-sample statistics.

Backtesting implicitly assumes that the future will resemble the historical period tested, an assumption that does not fully capture how dynamic financial markets can be, including upside and downside outcomes that have simply never yet occurred in the sample. Simulation is the complementary technique: rather than replaying history once in order, it explores a wider range of hypothetical outcomes. There are two broad approaches. Historical simulation draws returns at random from many historical periods without regard to their original order, which makes it, in effect, a non-deterministic version of rolling-window backtesting; it is widely used in market risk analysis at banks. Monte Carlo simulation instead assigns a statistical distribution to each key variable and draws random observations from that assigned distribution rather than from history directly, which lets the analyst incorporate features, such as fat tails or dependence between variables, that the raw historical sample may not fully represent. The cost is added complexity and heavier computation.

A generic eight-step process

Both simulation types follow the same skeleton. First, define the target variable, typically a portfolio’s return and its distribution. Second, specify the key decision variables, usually the return and weight of each underlying asset or factor, since the portfolio return is simply their weighted sum.

Third, choose the number of trials, N, typically between 1,000 and 10,000; more trials stabilise the estimated performance and its variance, though determining the mathematically optimal number is itself a non-trivial problem. Fourth, define the distributional properties of the key decision variables, which is where the two methods diverge: historical simulation draws from the historical record itself, while Monte Carlo simulation requires a specified statistical distribution for each variable. Fifth, use a random number generator to draw N random numbers per decision variable. Sixth, compute and store the target variable for each simulated draw. Seventh, repeat steps five and six until N trials are complete. Eighth, use the resulting N values of the target variable to compute the usual performance metrics, with CVaR and maximum drawdown the preferred downside measures for simulation output specifically.

Historical simulation, worked step by step

Historical simulation typically samples with replacement, known as bootstrapping, because the number of trials needed usually exceeds the number of historical observations available. Continuing with the eight-factor BM and RP portfolios, the target variables are their returns, the decision variables are the returns of the eight underlying factor portfolios (their weights are already known), and the simulation runs for N = 1,000 trials using bootstrapped monthly sampling. There are 374 months of factor return history, from April 1988 to May 2019, so each month is assigned an equal slice of the interval from 0 to 1, of width 1/374. A random number drawn from the uniform distribution is mapped to a specific month by dividing it by that width and rounding up to the next whole month.

Monte Carlo simulation, worked step by step

Monte Carlo simulation departs from historical simulation at exactly steps four and five. Instead of drawing from history, the analyst first fits a functional form to the factor returns, most commonly a multivariate normal distribution, chosen because it needs relatively few parameters: for K assets, the multivariate normal requires K mean returns, K standard deviations, and K(K−1)/2 pairwise correlations.

The model is calibrated on the same 374 months of history, April 1988 to May 2019, then random draws are mapped onto the fitted joint distribution to generate simulated factor returns for each trial.

Applying the same weighting logic as before, the Trial 1 factor returns give a BM portfolio return of 0.125 × (−3.2% − 3.1% − 0.2% + 0.7% + 2.3% − 3.3% − 1.7% + 1.9%) = 0.125 × (−6.6%) = −0.83%, and applying the May 2019 risk-parity weights gives an RP portfolio return of −0.86%. Repeating this across 1,000 trials produces the Monte Carlo return distributions for both portfolios.

Comparing the three approaches

Backtesting, historical simulation and Monte Carlo simulation are not competitors so much as cross-checks on one another, since each accounts for randomness differently. Across all three methods, the Sharpe ratio comparison between BM and RP is remarkably stable: RP outperforms BM by roughly the same margin regardless of which method produced the numbers. CVaR is far more sensitive to method. The multivariate normal Monte Carlo simulation understates BM’s downside risk relative to both rolling-window backtesting and historical simulation, because BM’s factor returns are negatively skewed with fat tails, exactly the features a normal distribution cannot represent. RP’s CVaR, by contrast, stays broadly similar across all three methods, because RP’s construction diversifies away much of the exposure to any single skewed factor, so the choice of distributional assumption matters far less for it than for BM.

Sensitivity analysis explores how much a target variable’s results change when a key input assumption is varied. Paired with simulation, its most natural use is testing whether the choice of statistical distribution materially affects the conclusions. The multivariate normal distribution used for the Monte Carlo simulation above is a reasonable starting point precisely because it needs so few parameters, but it cannot represent negative skewness or fat tails, both of which are visible in the factor return data. The natural next step is to refit the same eight factors to a distribution that can accommodate those features, a multivariate skewed Student’s t-distribution, and rerun the simulation.

The skewed t-distribution extends the multivariate normal by adding parameters for skewness and excess kurtosis, so it can track the empirical shape of the factor returns more closely, at the cost of being harder to estimate and requiring more parameters than the normal case. The procedure is otherwise identical to the Monte Carlo simulation already described: only step four, the distributional assumption, and step five, the draw from that distribution, change.

Trial 1’s equal-weighted BM return is 0.125 × (2.0% + 0.3% + 1.7% + 3.1% + 2.0% + 0.9% + 0.2% − 0.5%) = 0.125 × 9.7% = 1.21%. The corresponding RP return, applying the same May 2019 risk-parity weights used throughout, is 0.75%.

Widening the comparison to include the skewed t-distribution alongside backtesting, historical simulation and the normal Monte Carlo simulation, the Sharpe ratio comparison between BM and RP again barely moves; RP outperforms BM under every one of the four methods. CVaR tells a more layered story. For BM, the historical simulation and rolling-window backtesting CVaR figures resemble each other closely, and the two Monte Carlo variants, normal and skewed t, also resemble each other, but both Monte Carlo variants understate BM’s downside risk relative to backtesting and historical simulation. Adding skewness and kurtosis parameters to the distribution helped less than might be expected. That finding is itself informative: it suggests further sensitivity testing with still other functional forms would be worthwhile before relying on any single Monte Carlo specification for BM’s tail risk. For RP, the estimated return distribution looks fairly similar across all three simulation methods, because RP’s own returns are close to symmetric with only modest excess kurtosis, so there is comparatively little tail shape left for a richer distribution to capture.