MR 4 – Backtesting VaR

A value-at-risk figure is a forecast, and it earns trust only when its record holds up. Backtesting is the formal statistical procedure that supplies that record: the risk manager lines up every past VaR number beside the portfolio return that followed it and asks whether the realised losses behaved the way the model said they would. These reality checks are one strand of model validation, alongside stress testing and independent review.

The quantity at the centre of the exercise is the exception, also called an exceedence: a day on which the actual loss was larger than the VaR reported the evening before. A 99 percent daily VaR is a promise that losses beyond it will be rare, about one day in a hundred. So, if the model is correctly calibrated, the share of exception days in a long sample should settle close to 1 percent, and the whole question of backtesting becomes whether the observed count is close enough to that target.

Too many exceptions and too few exceptions are both failures

When exceptions arrive far more often than the confidence level allows, the model is understating risk. Trading desks then receive too little capital for the risks they run, and a regulator may add a penalty. The opposite outcome is less dramatic but still a fault. A model that almost never records an exception is overstating risk, which ties up capital that could have been deployed elsewhere and makes the allocation of capital across business units inefficient.

Backtesting carries particular weight in banking supervision because the Basel Committee allowed banks to use their own internal VaR models to set market risk capital. That permission was only defensible with a disciplined way of checking the models, since a bank that sets its own capital has an incentive to report a low VaR. So, a good backtesting framework should give a high chance of catching a bank that understates its risk on purpose, while not punishing a bank whose honest model was beaten by a run of bad luck. Balancing those two goals is the core statistical problem of this whole topic.

A historical example of calibration

Bankers Trust once published a chart that plotted the absolute value of each day’s profit and loss against its 99 percent daily VaR, with a 45 degree line separating the two. Any point above the line was a day on which the P&L, whether a gain or a loss, was larger in size than the VaR. Because the chart folds gains and losses together, the relevant tail is two sided: under a symmetric P&L distribution about 2 percent of days should land above the line, which is roughly 5 points over a year of trading. The bank showed four such points, which looks like good calibration. An accurate model could still have produced more in an unlucky year, so the real question is at what count the model should be rejected.

The VaR number and the realised return do not describe quite the same portfolio. A VaR forecast is computed on the positions held at the close and assumes those positions stay frozen over the horizon. Real trading books do not stay frozen. Traders buy and sell during the day, and the reported P&L also carries items that have nothing to do with market moves on the overnight positions, such as fees, commissions, bid and ask spreads earned, and net interest income.

The realised P&L is therefore contaminated relative to the forecast. A short horizon limits the contamination, which is why backtests are usually run on daily data. Even then, two effects pull in opposite directions. Intraday trading tends to raise the volatility of revenues, partly because traders often scale down positions toward the end of the day, so the closing book understates the risk carried during the session. Fee income works the other way, adding a steady positive stream to P&L that the VaR model never attempted to capture.

Three ways to measure the return being tested

A pure model test should use the hypothetical return, since that is what the forecast describes. Actual returns still matter because they are the real gains and losses regulators scrutinise. Supervisors under the updated market risk rules may accept either hypothetical or cleaned returns for this purpose.

The best practice is to backtest against both series and read the two results together. If the model passes with hypothetical returns but fails with actual returns, the modelling itself is sound and the trouble lies in intraday trading activity that the closing snapshot never sees. If the model fails even with hypothetical returns, the methodology and its assumptions need to be reexamined.

The simplest verification tool is the failure rate. Suppose a bank reports VaR at a left tail probability of p, where p = 1 minus c and c is the VaR confidence level, for T consecutive days. The analyst counts N, the number of days on which the loss exceeded the previous day’s VaR, and the failure rate is N/T. For a correctly specified model this ratio should be an unbiased estimate of p and should move closer to p as the sample grows.

No exception is ever expected to hit the target exactly. At a 95 percent confidence level an accurate model will sometimes show 8 percent exceptions purely through bad luck; if the rate reached something like 20 percent, the analyst would conclude the model, not luck, was at fault. The test formalises where between those two the line should fall. Keep two confidence levels apart: the VaR level c, such as 99 percent under the Basel rules, and the unrelated level of the test that judges the model, often 95 percent.

Exceptions as Bernoulli trials

Each day either produces an exception or not, and if the model is right each does so with probability p, independently. Such days are Bernoulli trials, so the count x over T days follows a binomial distribution.

A valuable feature of this test is that it is fully nonparametric. It assumes nothing about the shape of the return distribution, which could be normal, skewed, heavy tailed or changing over time; all that matters is whether each day was an exception. When T is large the central limit theorem lets the binomial be replaced with a normal approximation, which gives a quick z statistic.

With a two tailed test at a 95 percent test confidence level, the model is rejected when the absolute value of z is greater than 1.96.

Any rule of the form “reject the model if the count is above some cutoff” can go wrong in two distinct ways, and the cutoff decides how often each one happens. The table pairs the true state of the model with the decision taken.

A Type I error rejects a model that is in fact correct, so an honest bank is penalised for bad luck. A Type II error accepts a model that is in fact wrong, so an understated risk goes undetected. The ideal test holds the Type I error rate low and also delivers a very low Type II error rate; a test that achieves the second is called powerful, since power is the probability of rejecting an incorrect model, equal to 1 minus the Type II rate.

Seeing both errors in one picture

Take one year of data, T = 250, and a 99 percent VaR, so p = 0.01. If the model is right, the expected count is 2.5, and the binomial distribution in the top panel of Figure 1 shows how the count spreads around that average. Under a rule that rejects at five or more exceptions, a correct model still lands in the rejection region 10.8 percent of the time, which is the Type I error rate.

Now suppose the model is wrong and losses beyond the reported VaR really occur 3 percent of the time, so p = 0.03 instead of 0.01. The bottom panel shows that the count shifts to the right, centred near 7, yet with the same cutoff the incorrect model still shows four or fewer exceptions 12.8 percent of the time and escapes rejection. That 12.8 percent is the Type II error rate.

The usual approach is to fix an acceptably low Type I rate first and then look for the test design that makes the Type II rate as small as possible.

Kupiec (1995) recast the failure rate test as a log-likelihood ratio, often called the proportion of failures test. The statistic compares how likely the observed count is when the tail probability is fixed at the model’s p with how likely it is when the probability is set to the observed rate N/T, which is the best possible fit to the data.

When T is large and p is the true probability, LR_uc follows a chi-square distribution with one degree of freedom. So, at a 95 percent test confidence level the null is rejected when LR_uc is greater than 3.841. This is the same decision as the z test of the previous section in large samples, because a chi-square variable with one degree of freedom is the square of a standard normal variable, and 1.96 squared is about 3.841.

Nonrejection regions by sample size

Kupiec also tabulated, for a 95 percent test confidence level, the range of exception counts within which the null cannot be rejected. The table below keeps every region as published and adds the expected count pT for each sample length, which shows that each band sits around its expected value.

Source: adapted from Kupiec (1995). The pT columns are added here for comparison.

Reading one row shows how the table is used. With two years of data, T = 510, and a 99 percent VaR, the expected count is 1 percent of 510, about 5. The analyst cannot reject the model while N stays inside 1 < N < 11. A count of 11 or more says the VaR is too low and large losses are more likely than the model claims, while a count of 1 or fewer says the model is overly conservative.

The width of the band, measured as a proportion N/T, narrows as data accumulate. In the p = 0.05 row the band runs from 6/252 = 0.024 to 20/252 = 0.079 with one year of data, but only from 37/1000 = 0.037 to 65/1000 = 0.065 with 1000 days. More data make it easier to reject a model that is wrong.

Why a high VaR confidence level is hard to verify

The table also exposes an awkward fact. When p is small, deviations are very hard to confirm. For p = 0.01 and T = 252 the region is simply N < 7, with no lower bound, so even zero exceptions cannot be distinguished from correct calibration and there is no way to tell whether the model systematically overstates risk. Exceptions at the 1 percent level are simply too rare. This is why some banks backtest a VaR at a lower level such as 95 percent, which produces enough exceptions to test, and then apply a multiplicative factor to turn that figure into a capital cushion.

The Basel Committee’s (1996a) backtesting rules for the internal models approach are built directly on the failure rate test, but regulators work under constraints that a bank’s own risk team does not face. Supervisors cannot see inside each model and need rules that are comparable across institutions.

Designing such a test starts with the Type I error rate, the chance of rejecting a bank whose model is correct. That bank has simply been unlucky and should not be penalised unduly, so the rate should be low, perhaps around 5 percent depending on what a false rejection costs. The unavoidable conflict is that any test generous enough to spare honest banks will also commit Type II errors and let through some banks that deliberately understate their VaR.

The zones and the capital multiplier

The procedure records daily exceptions of the 99 percent VaR over the most recent year of 250 trading days. A correct model would average 1 percent of 250, or 2.5 exceptions. Up to four exceptions are accepted and place the bank in the green zone. Five or more move it into the yellow or red zone, where the multiplicative factor k applied to VaR in the capital charge rises progressively from 3 toward 4.

The resulting k column adds each increase to the base multiplier of 3.

A red zone result triggers the penalty automatically. Inside the yellow zone the penalty is left to the supervisor, who weighs the reason behind the exceptions. The Committee sorts those reasons into four categories.

The wording is deliberately loose, and exceptions caused by sudden abnormal market moves, major political events or natural disasters may even be excluded. Supervisors keep the freedom to adjust the rules in turbulent times.

The hardest part of supervisory backtesting is telling bad luck apart from a faulty model, which is the same thing as weighing Type I against Type II errors. The table sets the probability of each exception count for a correct 99 percent model beside the probability for an incorrect model whose true coverage is only 97 percent, both over T = 250 days.

Values as published. In the N = 10 row the power entry of 21.1 does not equal 100 minus 77.9; the binomial value is 22.1.

With the yellow zone starting at five exceptions, the Type I error rate is 10.8 percent, so roughly one bank in ten could be penalised although its model is correct. The Type II side is worse: a bank whose true coverage is 97 percent still passes with four or fewer exceptions 12.8 percent of the time. The framework is therefore not very powerful, and the miss is not a technicality. Under a normal distribution the 99 percent quantile is 2.326 standard deviations against 1.881 at 97 percent, so the true VaR of such a bank would be 23.7 percent larger than the figure it reports.

Two ways to raise power

The weakness comes from the very high 99 percent confidence level, which yields too few exceptions to test reliably. Suppose instead the VaR is set at 95 percent, with a larger multiplier k so that capital is unchanged. A correct model now averages 12.5 exceptions over 250 days, close to 13 a year. To keep the Type I rate near the Basel figure of 10.8 percent, the model is rejected at 17 or more exceptions, which corresponds to a Type I error of 12.5 percent. Assume, as before, that the true VaR is 23.7 percent above the reported one, which at the 95 percent level means true coverage of only 90.8 percent. The Type II error rate then drops to 7.4 percent. So, a 95 percent VaR makes a bad model far harder to miss.

The second route is more data. Keeping the 99 percent VaR but observing T = 1000 days, the cutoff would be set at 14 exceptions, for a Type I rate of 13.4 percent and a Type II rate of only 0.03 percent. More observations improve the test dramatically.

Every test so far checks unconditional coverage: whether the total count is right, ignoring when the exceptions happened. Timing matters too. At a 95 percent VaR a year should bring about 13 exceptions, spread fairly evenly through the calendar. If 10 of those 13 arrived within the last two weeks, the annual count could look acceptable while something has clearly gone wrong, perhaps a rise in market volatility that the VaR has not picked up, or traders building unusual positions that open risk holes. A sound system should also test conditional coverage, meaning coverage given current conditions.

Building the exception table

Christoffersen (1998) extended the Kupiec statistic by adding the requirement that exceptions be serially independent. Each day receives an indicator of 0 if VaR was not exceeded and 1 if it was. T_ij counts the days in state j that followed a day in state i, and π_i is the probability of an exception given state i on the previous day. The counts are arranged as follows.

If an exception today does not depend on yesterday, the two conditional columns should show the same exception rate, so π₀ and π₁ should both match the overall rate π.

The first term is the maximised likelihood when exceptions are independent across days, which forces a single probability π = π₀ = π₁ = (T₀₁ + T₁₁)/T. The second is the maximised likelihood with the two conditional probabilities estimated freely from the data. Joining this with the Kupiec statistic gives the full conditional coverage test.

Asymptotically each component is chi-square with one degree of freedom and they are independent, so their sum is chi-square with two degrees of freedom. At a 95 percent test confidence level the combined test rejects when LR_cc exceeds 5.991, and independence on its own is rejected when LR_ind exceeds 3.841.

Low power, worst with a high confidence level and a short sample, has prompted a search for better tests. The catch is that statistical decision theory shows the exception test to be the most powerful one available within its class. Any real improvement must therefore test a different hypothesis or bring in more information than a single quantile per day.

Extensions that use more information

Crnkovic and Drachman (1996) proposed a test of the whole forecast distribution based on the Kuiper statistic. It is still nonparametric and has more power, but it needs more than the VaR figure at one confidence level. A second route looks at the time between exceptions, known as duration, and Christoffersen and Pelletier (2004) showed that duration-based tests can beat the standard test when risk varies over time. A third uses parametric information: when VaR is a multiple of a forecast standard deviation, the forecast volatility can be compared directly with realised volatility, and the extra information raises power. A regulator, however, typically receives only the daily VaR, which is why exception counts remain the standard tool.

What bank data show

Berkowitz and O’Brien (2002) carried out the first empirical study of internal VaR models, using figures that U.S. banks reported to their regulators. The P&L distributions were generally symmetric but had fatter tails than the normal. Stahl and coauthors (2006) found that even strongly nonnormal trading positions aggregate to a roughly normal distribution at bank level.

For six U.S. commercial banks with roughly 600 daily observations each, a 99 percent VaR should have produced about 6 exceptions on average. The striking result was how few exceptions there were, even across the turbulent 1998 period. Bank 4 recorded zero, and its VaR was several times larger than its biggest P&L swings. Across banks 3 to 6, average VaR was at least 60 percent above the actual 99th percentile of P&L. In other words the banks reported conservative VaR figures and so carried market risk charges, and regulatory capital for trading, that were higher than their risks required.

Several explanations are possible. Backtesting on actual rather than hypothetical returns lets fee income lift P&L and remove exceptions. The models may be too simple, for example by ignoring diversification across positions. Capital requirements may not be binding, since U.S. banks held economic capital above the regulatory minimum, so reporting a high VaR cost little and kept supervisors at a distance. A simpler story is the chief executive who calls in the risk officer after every exception: after hearing that 2 to 3 exceptions a year are normal at 99 percent, and still being summoned, the risk officer quietly raises the confidence level until exceptions stop.

Choosing backtesting parameters

Keep the horizon short, which multiplies observations and limits the effect of changing positions, and do not set the confidence level so high that the test is starved of exceptions. Portfolios and models also change over time, adding structural instability to any long record. Even so, a well designed backtest gives continual feedback and stops a risk model from drifting away from reality.