MR 6 – Validating Bank Holding Companies’ Value-at-Risk Models for Market Risk

Value-at-Risk entered bank capital rules in 1996, when the Basel Committee on Banking Supervision allowed banks to capitalise their trading books with their own VaR models. The same rules came with a check: every day the bank compares the loss it actually suffered with the VaR it forecast the day before, and a loss larger than the forecast counts as an exception. The tally of exceptions over the past year feeds a stoplight test built on the work of Kupiec (1995), and the more exceptions a bank records, the larger the multiplier applied to its VaR when its capital requirement is set. Because capital depended directly on it, backtesting became the dominant way banks thought about validating their market risk models.

That emphasis is understandable, and backtesting will keep its place, but it has crowded out other parts of validation that deserve equal attention. The academic literature has also moved on, producing sharper backtests and new tools for measuring the precision of a VaR number, and those tools can be used inside banks today. The Basel Committee has since moved its capital framework to expected shortfall (BIS 2019), with implementation expected in 2025, yet the validation ideas in this lesson carry over to that measure with little change.

The three parts of model validation guidance

US banking agencies have issued model validation guidance that applies to every model a bank uses, trading models included. The Basel backtesting rules are simply a more detailed requirement on one corner of that broader guidance. The guidance asks for three kinds of work:

• Conceptual soundness. The validator reviews the data, the assumptions and the techniques inside the model, and runs sensitivity analysis on the parts of the model where it is warranted.
• Outcomes analysis. Forecasts are compared with what actually happened, and backtesting is the natural form of this for a VaR model.
• Benchmarking. The model is compared with one or more alternative models to see whether it holds up against a credible competitor.

Banks have historically put almost all their effort into the second item for trading. The lesson works through all three in turn, first setting out what the common VaR models look like, then showing how each validation step applies to them, and finally reporting how the VaR models of US bank holding companies fared when backtests and benchmark comparisons were actually run on their data.

VaR is a statement about the left tail of the distribution of changes in portfolio value. Using what is known at time t, collected in the information set written as F with subscript t, the probability that the portfolio loses more than VaR between t and t + N equals 1 − c, with c denoting the coverage level:

The 1996 Basel rules used a horizon N of 10 days and a coverage level c of 99%, so the superscript on VaR in that setting is 1%. Two conventions are worth fixing early. A VaR model is normally described by its coverage, the probability that the institution does not see a loss larger than VaR, rather than by the exceedance probability 1 − c. And VaR is quoted as a positive figure although it measures a loss, which is why a minus sign sits in front of it in the definition.

Building the change in portfolio value from positions

Portfolio value changes are additive across positions: each position i in the portfolio P contributes its own change:

Most VaR models need a pseudo history of portfolio value changes. The usual construction takes one-day changes, so N = 1, and holds the composition of the portfolio fixed at its current values V with subscript iT at time T. Each historical date t then supplies a return r with subscript it for every position, and the pseudo history asks what today’s portfolio would have gained or lost on each of those past days:

Building this series is far from trivial. Every position needs a usable return history, positions without one need proxies, and pricing approximations creep in. Supervisory reviews spend a great deal of time checking that the pseudo history is accurate, because every downstream VaR number inherits its errors.

Why profit and loss rather than returns

Academic studies usually model returns or log returns. Practitioners work with profit and loss, the dollar change in portfolio value, and the analysis in this lesson follows practitioners so that it applies directly to data banks actually report. The cost of that choice is statistical. Every test discussed later assumes that the variable being tested is independently and identically distributed, or has been transformed to be so. When a bank turns over its trading book frequently, its raw P&L is not i.i.d. because the positions generating it keep changing, and using the P&L without transformation breaks that assumption.

Once the pseudo history exists, the simplest route to a VaR is to treat it as the distribution of tomorrow’s value change and read off the right quantile. This is historic simulation (HS):

In practice the T value changes are sorted from the worst loss to the best gain and the observation sitting at the (1 − c) quantile is selected, with interpolation between neighbouring observations when the rank is not a whole number. Historic simulation remains the most widely used VaR method at commercial banks despite two well-known criticisms. Pritsker (1997) showed that it responds sluggishly to recent changes in volatility, since a calm day and a turbulent day from years ago carry the same weight. Escanciano and Pei (2012) criticised the practice of backtesting historic simulation models with unconditional tests.

GARCH VaR and the RiskMetrics special case

To make the model react to recent conditions, many banks let the volatility of portfolio value changes depend on the recent pseudo history through a GARCH model. Each change is a conditional mean plus a scaled shock, and the variance updates from past squared changes and past variances:

RiskMetrics (1997) is the best-known special case. It assumes g is the standard normal distribution and replaces the estimated GARCH recursion with a restricted ARCH(1,1) form driven by a single decay factor λ:

Filtered historic simulation

The weakness of any GARCH VaR is that the shock distribution has to be specified. Filtered historic simulation (FHS), due to Barone-Adesi et al. (1999), keeps the GARCH volatility but drops the distributional assumption. The pseudo history is divided by its fitted volatilities to recover standardised shocks z, those shocks are sorted, and the empirical (1 − c) quantile of the shocks is rescaled by tomorrow’s forecast volatility:

So FHS captures volatility clustering through the GARCH filter while letting the data decide how fat the tail of the shocks is. All three methods described so far are univariate, treating the portfolio as one series. A more disaggregated multivariate model splits the pseudo history into its individual positions and estimates each position’s volatility plus every pairwise correlation, which quickly becomes intractable as the number of positions grows. Andersen et al. (2006) and Christoffersen (2012) show how to extend the univariate methods in that direction.

A large VaR model embeds hundreds of design decisions, from how positions are mapped to risk factors to how missing prices are filled. The conceptual soundness review asks whether those assumptions, techniques and data are appropriate and how realistic each choice is, and it is usually written up as a narrative by the validator. Underneath all the detail sits one basic question: is the model suitable for the purpose it was built for? For a VaR model the primary purpose is helping the firm control the risk in the positions it holds. Capital calculation is another use, but it is arguably secondary to risk management.

The trap of a VaR built on actual P&L

Berkowitz and O’Brien (2002) compared the VaR models of large banks with a simple alternative: a GARCH model fitted directly to each bank’s actual daily profit and loss. The simple model forecast better than the banks’ own models. It is tempting to read that as a reason to replace position-based VaR with GARCH on actual P&L, and that conclusion would be wrong.

An exchange between Lo (2001) and Jorion (2007) shows why. Lo described a hypothetical hedge fund, Capital Decimation Partners, whose strategy was to sell out-of-the-money put options and adjust the book dynamically. Judged by its realised returns, the fund looked like a source of attractive profits with little risk, because risk measures computed from its track record could not see how the dynamic strategy was positioned. Jorion showed that once the VaR used positional information, rebuilding the pseudo history whenever the positions changed, the measured risk rose sharply and tracked the moments when the book became more dangerous. A P&L-based measure looks backwards at what the fund used to hold; a positional measure looks at what it holds now.

The fit-for-purpose test

The same point shows up in daily risk management. Suppose a risk manager tells a trader to cut the VaR of a book, and the trader dutifully reduces the positions. If VaR is computed from the book’s actual P&L history, nothing happens to it, because the history of past profits and losses is unchanged by today’s trades. A VaR built on the pseudo history falls straight away, because the position values V with subscript iT that drive it have changed. So the key purpose of a VaR model is to show how risk moves when positions move, and a model that cannot do this is not fit for purpose and fails a central conceptual soundness test.

Moving from narrative to quantitative checks

Beyond the question of use, conceptual soundness can be probed with quantitative work. Sensitivity analysis shows how much data limitations and modelling choices move the VaR, and statistical confidence intervals show how precisely the VaR is estimated in the first place. Together they reveal how serious the data problems and estimation errors are, and they tell the validator how far the model can be trusted when it is used to manage positions. The next three sections develop these tools.

If the purpose of VaR is to show how risk responds to positions, then the natural sensitivity analysis measures how VaR responds to each position. Garman (1997), Gourieroux et al. (2000), Tasche (2000) and Hallerbach (2003) developed ways to decompose VaR and study its sensitivities. For validation the main payoff is a way to test the simplifications and omissions made while building the pseudo history. Supervisors increasingly ask banks to track and quantify the risks not in VaR and to keep a record of where data proxies are used. Those exercises are usually ad hoc, and sensitivity analysis gives them a consistent framework.

Euler decomposition, marginal VaR and component VaR

VaR is linear homogeneous in positions: scale every position by a factor and VaR scales by the same factor. That property means VaR satisfies the Euler equation:

The derivative is the marginal VaR, which says how VaR would change for a small change in the size of position i, and each term in the sum is that position’s component VaR. The marginal VaR is tied to the slope of a regression of the position’s value change on the whole portfolio’s value change:

The component therefore depends on two things: the position’s weight in the portfolio, w with subscript i, and how strongly the position’s value moves with the whole portfolio. If the bank is uneasy about how one position is valued, the decomposition shows how much the VaR could be affected by that choice.

When the regression cannot be run

The regression route breaks down in exactly the cases a validator cares about. Equation 6.5 needs a full time series of the position’s value changes, yet the reason a bank wants to know the position’s impact is often that this valuation data is missing or scarce. A proxy can fill the gap, but a proxy may be less volatile than the real position or less correlated with the portfolio, and either flaw understates the position’s contribution to VaR.

Tasche (2000) and Hallerbach (2003) offer an alternative that is especially simple under historic simulation. Linear homogeneity implies that the VaR scenario can be split into the expected change of each position, conditional on the portfolio losing exactly its VaR amount:

Under historic simulation the VaR comes from one particular day in the pseudo history, so the component of any position is estimated by what that position would have done on that day. For a missing position, the problem reduces to asking what loss it would have suffered on the VaR day. One day is a noisy estimate, so averaging the position’s value changes across the ordered observations around the HS VaR gives a steadier figure.

Benefits and challenges of the approach

There is an awkward circularity here. If a bank had enough data to estimate an omitted risk accurately, that risk would probably already be in the VaR; when it is missing, the data is thin and any estimate of how much it matters is imprecise. Even so, the method is general. A valuation built on a truncated Taylor series, for instance, can have its first omitted higher-order term assessed in the same way.

The answer also depends on what else is in the book, and a large trading portfolio changes fast. For that reason many supervisors ask for risks not in VaR to be measured both as a component VaR and as a stand-alone VaR, giving a view of the omission that does not hinge on today’s portfolio mix. The whole exercise fits the validation idea set out by Jarrow (2011): test whether the model is sensitive to its assumptions, here the assumption that omitting a position or risk factor is immaterial. It also gives the bank a disciplined way to rank which risk factors or higher-order terms should be added to the model first.

A VaR figure is an estimate from a finite sample, so a natural follow-up is how accurate it is. The statistical answer is a confidence interval that contains the true VaR a stated, high percentage of the time, which measures the estimation risk in the number. In most statistical work such an interval is routine. Most financial institutions nevertheless do not compute one for their VaR, which is odd for a model that is statistical through and through, and it is a test that should be part of routine validation.

Jorion’s asymptotic standard error

Jorion (1996) gives the asymptotic standard error of a quantile estimate, which can be applied to VaR:

VaR, T and c are known, so the only hard part is the density f evaluated at the VaR. Imposing a distribution is the obvious route, but financial returns are not normal, so assuming normality is probably a mistake. Jorion also notes that it may be better to base the interval on the variance of the whole distribution of pseudo returns rather than on a single quantile point. The approach stands or falls on how f is determined.

Order statistics

Two non-parametric routes avoid choosing f. The first, from Dowd (2006), uses order statistics. Sort the pseudo history from lowest to highest. The chance that r or more of the T observations fall at or below a value ΔV is a binomial sum in F(ΔV), the cumulative distribution of value changes:

Since minus the VaR is simply one value of ΔV, the same formula can be evaluated there. For a 90% interval, G is set to 0.05 and to 0.95 and each equation is solved numerically for F, giving F with subscript l for the lower bound and F with subscript u for the upper bound. For historic simulation the interval is then read off the sorted pseudo history:

For a GARCH VaR, the estimated variance and the assumed distribution define F, and the ΔV values matching the two bounds are taken from that function.

Bootstrapping

The second route is the bootstrap of Christoffersen and Goncalves (2005), with versions for historic simulation, GARCH(1,1) and FHS. These procedures are straightforward for univariate models, though they can be computationally heavy. For historic simulation, draw T observations with replacement from the pseudo history to form a bootstrap sample S with subscript i, and compute its VaR:

Repeating this B times gives B VaR values that trace out the sampling distribution, and a 90% interval is the 5th and 95th percentiles of the ordered bootstrapped VaRs. For a general confidence level CI, the bounds are the quantiles that leave (1 − CI)/2 in each tail:

Like historic simulation itself, this bootstrap is non-parametric, but it assumes value changes are independent over time, so any dependence in returns is lost. For a GARCH VaR the resampling must respect the dependence, and the easy way is to resample the i.i.d. shocks z from Equation 6.1 and rebuild bootstrap paths of ΔV and σ through Equations 6.1 and 6.2. Computing the VaR from Equation 6.3 with the original parameters would ignore estimation risk, so Equation 6.2 is re-estimated on each bootstrap path, giving new volatility estimates and a recomputed VaR:

The B recomputed GARCH VaRs are then ordered and the interval read off exactly as for historic simulation. For FHS the parametric quantile in Equation 6.7 is replaced by the re-estimated residuals, and each bootstrap VaR is computed from Equation 6.4. More recent work builds intervals from the empirical likelihood: Chan et al. (2007) for GARCH models and Gong, Li and Peng for ARCH models.

Applying these methods to S&P 500 returns gives a feel for how precise a one-day 99% VaR really is. The comparison covers three VaR methods (historic simulation, GARCH and FHS), two interval methods (bootstrap and order statistics) and four windows: a recent year (N = 250), a long history (N = 9424), a one-year stress window (N = 253) and a three-year stress window (N = 755). The table below reorders the results by window and adds a derived column, the total width of the interval as a percentage of VaR.

Below VaR % is (VaR minus lower bound) divided by VaR; above VaR % is (upper bound minus VaR) divided by VaR. The width column, computed here, is (upper bound minus lower bound) divided by VaR, so in a few rows it differs by 0.1 from the sum of the two rounded percentages. The text describes these as 90% intervals while the original column heading reads 95%CI.

Every printed percentage reconciles with its VaR and bounds. Four lessons follow. First, the intervals are not symmetric around the VaR: for historic simulation in the recent year, the order statistics interval reaches 21.3% below VaR but only 2.1% above it. Second, more data generally tightens the interval, with the HS width falling from 36.7% on 250 days to 14.3% on 9424 days, although the pattern is not universal, since the GARCH interval is wider on the long sample (31.4%) than on the recent year (27.6%). Third, for historic simulation neither bootstrap nor order statistics gives consistently tighter intervals; each wins in some windows. Fourth, historic simulation intervals become very wide in stress periods, while FHS intervals are strikingly narrow everywhere, which points to FHS as the more efficient estimator.

Intervals on bank P&L

Banks’ own pseudo histories are not public, so, following Berkowitz and O’Brien, two VaRs were computed on each bank’s trading P&L, meaning the change in value of positions held at the previous day’s close. Such a VaR is unsuitable for risk management, as the conceptual soundness discussion showed, but it does describe how precise VaR estimates built on bank P&L can be.

Transposed from the original layout. The original caption describes the P&L as hypothetical.

The median bank’s HS interval sits 19.60% below and 15.92% above its VaR, while the median GARCH interval is only 1.20% below and 2.30% above. The evidence is not conclusive, but the much narrower GARCH intervals, read together with the S&P 500 results, suggest that banks not already using filtered historic simulation could get more precise VaR estimates from it. Few bank holding companies compute VaR confidence intervals routinely, which is puzzling given that the methods are well established and the accuracy information is valuable.

Outcomes analysis compares what the model predicted with what happened, and for VaR that comparison is backtesting. Banks have done it since 1996, when the Basel Committee wrote it into capital requirements. Early studies of the reported results, summarised by Berkowitz and O’Brien (2002), found the models lacking: losses beyond VaR were rare and they arrived in clusters, a sign of dependence. Perignon and Smith (2006, 2010a, 2010b) found the same pattern and offered explanations for why the models were so conservative. O’Brien and Szerszen (2017) found bank VaR models conservative before and after the financial crisis but not during it.

Which P&L should be compared with VaR

All this evidence depends on what banks report, and most banks historically reported actual P&L. Actual P&L contains items a VaR model never tries to capture, such as fee income and commissions. Fresard, Perignon and Wilhelmsson (2011) simulated the effect of these contaminating items and showed they could help explain why backtests made VaR look conservative. More recently, US bank holding companies under the Basel 2.5 trading rules have reported, at sub-portfolio level, the P&L the bank would have earned had it frozen the previous day’s closing positions for one day. That is much closer to what VaR measures. Fees and commissions are not in the model, so they should not be in the P&L used to test the model, although much published research on actual backtesting still includes them.

The exceedance indicator

The basic observation behind backtesting, developed by Kupiec (1995) and Christoffersen (1998), is that a 99% VaR should be breached on 1% of days. The first step is an indicator that records each breach:

Unconditional coverage asks whether breaches happen with probability 1 − c on average, ignoring what was known beforehand. Conditional coverage asks the stronger question of whether the breach probability equals 1 − c at every point in time, given everything known at t:

The likelihood ratio test

The null hypothesis of the unconditional test is that the indicator is an i.i.d. Bernoulli variable. Let T with subscript 0 and T with subscript 1 count the days with no breach and with a breach, summing to the sample size T. The likelihood of a breach probability π is:

The observed breach rate T with subscript 1 divided by T is compared with the rate 1 − c that the model promises, through a likelihood ratio that is chi-squared with one degree of freedom:

The example shows the low power that Kupiec himself documented: with only a year of daily data, a model breached more than twice as often as it should be sits right on the edge of rejection. Large samples are needed to reject models that are wrong, and the tests in the next two sections try to address this by bringing more information into the test.

A model can have the right number of breaches on average and still be dangerous if those breaches bunch together. A cluster means that, during certain stretches, extreme losses are much more likely than the VaR suggests, which is precisely when a bank can least afford it. If breaches were independent over time there would be no clustering to worry about. Christoffersen (1998) built a test of independence and combined it with the coverage test into a joint test of conditional coverage.

The Markov chain model of breaches

Christoffersen models the breach indicator as a first-order Markov chain. Today’s state j has a probability that depends only on yesterday’s state i, giving a transition matrix:

Independence means a breach yesterday does not change today’s breach probability, so the null hypothesis is that π with subscript 01 equals π with subscript 11, and both equal the overall breach rate T with subscript 1 divided by T. Let T with subscript ij count the days in state i followed by a day in state j. The maximum likelihood estimates and the unrestricted likelihood are:

The independence test says nothing about whether the breach rate is right. The conditional coverage test asks both questions at once, testing whether π with subscript 01 and π with subscript 11 both equal 1 − c, and its statistic is the sum of the other two:

Testing for longer memory

Pajhede (2017) generalises the conditional coverage test to dependence beyond one day. Allowing a full Kth-order Markov chain multiplies the parameters and soon becomes intractable, so Pajhede widens the window instead: π01 becomes the breach probability after K days with no breach, and π11 the breach probability when at least one breach occurred in the previous K days. That is one way to extend the dependence structure among many. For a general check of higher-order dependence, a Ljung-Box test can be run on the indicator, though it tests independence only and says nothing about coverage.

Regression-based backtests let the validator bring in any information that was available before each forecast and ask whether it would have predicted a breach. If a correct VaR has already used all relevant information, nothing known at t − 1 should help forecast whether day t is a breach.

The dynamic quantile test

Engle and Manganelli (2004) propose the dynamic quantile (DQ) test, built on a linear probability model. It is simple to implement, it allows for dependence in breaches, and it can test whether extra variables would improve the VaR. The dependent variable is the de-meaned indicator, Hit with subscript t equal to I with subscript t minus (1 − c), and X with subscript t collects variables known at t − 1, including lagged values of Hit:

The logistic dynamic quantile test

The breach indicator only takes the values 0 and 1, which makes a linear model awkward. Berkowitz et al. (2011), Patton (2006) and Clements and Taylor (2003) recast the DQ idea as a logistic regression, which is called the logistic dynamic quantile (LDQ) test. An nth-order autoregression for the indicator is:

Other variables known at t − 1 can be added. A likelihood ratio test checks the betas jointly, and a Wald test checks each one. If the betas are zero, the breach process is independent (including at orders above one) and the VaR has used the available information properly. Correct conditional coverage then requires the constant alone to deliver the right breach probability:

These regression tests are very easy to run, and Berkowitz et al. (2011) found the LDQ test to have significant power advantages over the other tests they compared.

The VaR quantile regression test

Gaglianone et al. (2011) take a different route and use quantile regression, extending the Mincer and Zarnowitz (1969) forecast evaluation regression to quantile forecasts. The value change is regressed, at the 1 − c quantile, on the VaR forecast:

A good VaR model has an intercept of zero and a slope of one. A non-zero intercept means the VaR is biased, consistently too high or too low. A slope greater than one means that high VaR values under-predict the quantile of value changes. Further variables known at t − 1 can be added, and all of them should have zero coefficients in a correct model. This VaR quantile regression (VQR) test uses the size of each value change rather than just whether a breach occurred, which is why it carries more information than indicator tests.

Choosing the significance level

Every test trades a type I error, throwing out a model that is correct, against a type II error, failing to reject a wrong one. Raising the significance level makes type I errors more likely and type II errors less likely. In risk management a type II error is costly, because a bad model stays in use, so it is reasonable to accept a lower confidence level than academic work would use. That is why a 90% confidence level is a sensible choice for VaR backtests.

Small samples and Monte Carlo p-values

Backtesting samples are rarely large and breaches within them are rarer still. Christoffersen (2012) and Berkowitz et al. (2011) recommend replacing p-values from the asymptotic distribution, usually chi-squared, with the Monte Carlo simulated p-values of Dufour (2006). Simulating the statistic under the null gives a correctly sized test in small samples, which is about as much as can be done about the small-sample problem. Low power remains the bigger issue, and bringing more information into the test, as the DQ, LDQ and VQR tests do, is the main way to raise it.

Other tests exist. Duration-based tests (Christoffersen and Pelletier, 2004) are largely unused in practice. Gordy and McNeil (2020) extend backtesting with probability integral transforms weighted so that the user can choose which parts of the distribution count most.

The tests above were run on the trading portfolios of US bank holding companies subject to the market risk rule. The P&L definition matters for interpreting the results. Before 2013, banks compared VaR with actual P&L, which included fees, commissions, intraday trading revenue and other items. These usually pushed P&L above the pure change in value of the previous day’s closing portfolio, and only occasionally below it. They can also hide real problems: a desk can be losing money on its inventory while fees and commissions make it look profitable. Current rules strip these items out, so the reported figure is the value change on the portfolio held at the previous day’s close, which is exactly what the VaR model tries to forecast. That is the right test of the model itself, although a supervisor might argue that the bank’s ability to earn other revenue also matters.

The data cover the second half of 2013 to 2016. Most banks were under the rule for substantially the whole period, while a few joined slightly later. Twenty bank holding companies reported backtesting data for at least 499 and up to 647 trading days, each giving a daily one-day 99% VaR and the matching P&L.

An average breach rate of 0.4% against the 1% a correct 99% model should show means the banks were, on average, conservative. Five banks recorded no breaches at all. Since 2013 to 2016 was a benign period for markets, this fits the familiar finding that bank VaR models run conservative in calm conditions. Not every bank did, though: the highest breach rate was 2.1%, more than double the expected rate.

Results of the individual tests

The tests were run as two-sided tests at 90% confidence, so a bank could fail for being too conservative as well as for being too aggressive, and no extra explanatory variables were included.

UC unconditional coverage, CC conditional coverage, DQ dynamic quantile, LDQ logistic dynamic quantile, VQR VaR quantile regression. No p-values are reported for VQR. The conservative row is failures minus non-conservative failures.

The four indicator-based tests behaved much alike: only two to four banks failed, and only one or two of those failures came from an aggressive model. The VQR test was in a different league, failing 19 of the 20 banks, 8 of them for being non-conservative. It is evidently far more demanding, perhaps also because it is more affected by the non-stationarity of portfolios that keep changing.

Why the indicator tests look kind

Indicator tests dominate practice, both because they are simple and because regulation uses them; the quantile test is included as a stricter point of comparison, and the models do poorly on it. One reading is that judging models by breach counts may have changed how banks build them, encouraging designs that pass the indicator test, possibly at the cost of performing well on other measures of accuracy.

Benchmarking is probably the most neglected part of market risk model validation. Many banks run a new VaR model alongside the old one for a short time to see that it behaves, but a formal comparison of the production model with a rival model is rare. That is understandable, since building one VaR model at a bank is already costly and time-consuming and building two doubles the burden. Yet whenever a bank replaces or upgrades its model there is a window in which both versions exist, and a formal comparison costs very little. The usual practice is to plot the two VaR series over time, which rarely shows more than that one is more conservative than the other or that they behave alike. Formal tests would be a real improvement, and the forecasting literature is well developed; Komunjer (2013) surveys the evaluation of quantile forecasts, VaR included.

Two obstacles and a ready-made benchmark

Two features of trading hinder statistical comparisons. Portfolios change frequently, so forecast errors are not i.i.d., a problem that bites hardest in regression-based tests. Fixes proposed by Christoffersen et al. (2001) work, but only for location-scale distributions where the quantile is a linear function of volatility. The second obstacle is that a bank rarely has two VaR models to compare. Berkowitz and O’Brien (2002) solve this by fitting a GARCH(1,1) VaR to the bank’s trading P&L, data every bank has. As the conceptual soundness section showed, such a model is unsuitable for risk management, but it is a convenient comparator and, as the evidence shows, a hard one to beat.

Loss functions

Formal forecast comparison starts with Diebold and Mariano (1995). Their paper did not deal with quantile forecasts specifically, but it laid the groundwork for comparing any kind of forecast, and one of its key points is that the evaluation should reflect the forecaster’s own loss function, which need not be the mean squared error used for point forecasts. Lopez (1996) proposed a regulatory loss function. Capital depends on VaR, so the regulator cares about losses that exceed VaR and much less about days when VaR exceeds the loss. Remembering that VaR is a positive number standing for a loss, the regulatory loss is:

Only breaches are penalised, so this criterion clearly rewards conservatism. For accuracy, the quantile can instead be scored with the same check function used to estimate a quantile regression, which penalises both breaches and excess cushion:

The sign test

Sarma et al. (2003) used the sign test of Diebold and Mariano to compare VaR models estimated on India’s NSE-50 index and on the S&P 500. The test asks about the median of the difference in losses between rival models i and j, Z with subscript t equal to the loss of model i minus the loss of model j:

Under the null of equal performance, the differential is positive half the time. Rejecting the null indicates that model i is significantly better under the chosen loss function when its losses are smaller on significantly more days. The test is simple enough that banks have little excuse not to make their comparisons formal.

Positional VaR against P&L VaR at US banks

The backtesting data allow each bank’s own model, the positional VaR, to be compared with a GARCH(1,1) VaR fitted to the bank’s P&L, the P&L VaR, both at one day and 99% coverage. Under the regulatory loss function the positional VaR wins significantly at every bank, which reflects how conservative the banks’ models are, a trait often attributed to regulatory oversight (Perignon, Deng and Wang 2008). Under the check loss, which rewards accuracy, the picture reverses: the P&L VaR wins at sixteen of nineteen banks, the positional VaR wins at one, and the remaining two show no significant difference. The banks’ models are built to be conservative, and that design choice appears to stop them beating the simpler benchmark on accuracy.

US regulators expect banks to test conceptual soundness, carry out outcomes analysis and benchmark their models, and the tools covered here turn those expectations into concrete procedures for VaR. Under the fundamental review of the trading book, BIS (2019), VaR is replaced by expected shortfall for regulatory capital, but most of the validation toolkit transfers directly to the new measure.

Both Dowd and Christoffersen and Goncalves show how to build confidence intervals for expected shortfall, so estimation risk can be measured just as it is for VaR. Hallerbach and Tasche give examples of measuring how missing risk factors affect expected shortfall, which carries the sensitivity analysis over. Du and Escanciano (2016) describe a relatively simple backtest for expected shortfall models. And the sign test can be applied to expected shortfall forecasts as well, so benchmarking against an alternative model remains straightforward.

The evidence from US banks gives the validator a clear agenda. Position-based models are the right foundation because only they respond to trades. Confidence intervals should be produced routinely, and the S&P 500 comparisons favour filtered historic simulation on precision. Indicator backtests are necessary but lenient, so stricter tests such as the quantile regression test belong in the toolkit. And a simple GARCH model on P&L, run as a benchmark under an accuracy-based loss function, is a cheap way to see whether conservatism is masking a lack of precision.