MR 7 – Beyond Exceedance-Based Backtesting of Value-at-Risk Models

Value-at-Risk is a loss threshold: for a chosen horizon and confidence level, it is the loss the portfolio should not exceed except on a known small fraction of occasions. Banks use VaR to set regulatory capital, to cap the risk individual traders may take, and to steer their own internal risk management. The regulatory version is a one-day VaR at 99% confidence. If a bank reports a one-day 1% VaR of $1,000,000, it is claiming that a one-day loss larger than $1,000,000 should happen on about one day in a hundred, and that on the other 99% of days the loss stays inside that figure. Put differently, when the model is right, every single day carries exactly a 1% chance that the next day’s profit and loss (P&L) shows a loss beyond the reported VaR.

A large literature studies how well such models behave. Jorion (2002) surveyed VaR specifications and called VaR the new benchmark for controlling market risk, Marshall and Siegel (1997) and Pritsker (1997) studied forecast performance on hypothetical portfolios, Berkowitz and O’Brien (2002) examined the regulatory VaR forecasts of large trading banks, and Berkowitz et al. (2016) looked at accuracy for a small set of trading desks. The study this lesson draws on adds two things: evidence on how bank VaR models fare at 99%, and tests of the whole forecast distribution applied both firm-wide and at the level of individual subportfolios.

Backtesting and the exceedance count

Backtesting is the statistical comparison of what a model forecast with what actually happened. For VaR, the standard practice since the mid 1990s has been to record every day on which the realized P&L falls below the VaR set the day before, which is the same as saying the loss exceeded the estimate. These days are called VaR exceedances, exceptions or breaches. A disciplined backtesting regime keeps a model honest both for internal risk control and for capital.

Christoffersen (1998) showed that a VaR model whose forecasts match the true P&L distribution must produce exceedances with two properties:

• Unconditional coverage. Over a sample, the share of days with an exceedance must match the stated tail probability, so a 99% one-day VaR should be breached on 1% of days. Kupiec (1995) built the first formal test of this property.
• Independence. Exceedances must not bunch together. Today’s breach should carry no information about whether tomorrow will also bring a breach.

Conservative and aggressive misspecification

When a model fails either property, the error usually leans one of two ways. A model with conservative misspecification forecasts a distribution that is too wide: realized P&L tends to land in the middle of the forecast, because the parameters assume more volatility than markets deliver. A model with aggressive misspecification forecasts a distribution that is too narrow, so realized P&L turns up in the tails far more often than the model allows. The first wastes capital; the second hides risk.

Counting exceedances has a built-in blind spot. It only asks whether the 1% quantile is right, so it says nothing about how the model performs at the 5% point, at the median, or deep in the 0.1% tail. The rest of this lesson builds tools that look at the whole forecast distribution.

A VaR model does more than publish one number; behind the VaR sits a full forecast distribution of tomorrow’s P&L, written F_t because it is built with information available on day t. Once the next day’s P&L, PL_t+1, is known, it can be fed back into that forecast distribution. The output is the probability the model assigned to a P&L at or below what actually happened. That probability is the probability integral transform, or PIT, introduced for density forecasts by Diebold et al. (1998) and often reported by banks as a “p-value”:

A PIT close to 0 means the realized outcome sat deep in the left (loss) tail of the forecast, a PIT near 0.5 means it landed in the middle, and a PIT close to 1 means it was a large gain relative to the forecast. Because every day’s outcome is placed on the same zero to one scale, PITs can be pooled across days even when the portfolio’s size and volatility change.

From the PIT back to the exceedance

The daily hit series records breaches of the VaR at tail probability p. Writing VaR as the p-quantile of the P&L distribution (a negative number for a loss), the indicator is

The two series carry the same information at the VaR level. P&L falls below the p-quantile exactly when the forecast CDF at that P&L is below p, so a hit at 1% occurs on precisely the days when the PIT is below 0.01. The hit series keeps only that yes or no answer; the PIT keeps the full location of the outcome, which is why it can be used to check the model at any percentile.

If the forecast distribution is correct, a PIT below 0.01 should happen on 1% of days, a PIT below 0.03 on 3% of days, and a PIT below any level u on a share u of days. That is the definition of a uniform distribution over zero to one, U(0, 1). So unconditional coverage of exceedances, which checks one point, generalizes into uniformity of the PITs, which checks every point at once.

The independence property generalizes in the same way. Exceedances should not cluster, and for PITs the requirement is that today’s PIT says nothing about tomorrow’s. Rosenblatt (1952) supplies the formal result: when the forecast distributions are exactly right, the sequence of PITs is independent and identically distributed U(0, 1). A test of model accuracy therefore becomes a test of whether the PIT series is i.i.d. uniform.

• Coverage at 1% maps to the share of PITs below 0.01; coverage at every level maps to uniformity of the PIT distribution.
• Independent exceedances map to PITs with no serial dependence at any point of the distribution.

This is why the two families of tests below are organized in parallel. Section by section, the exceedance tests (Kupiec, Christoffersen, Ljung-Box, logit regressions and duration) have PIT counterparts (Kolmogorov-Smirnov, Anderson-Darling, Cramer-von Mises and regressions of transformed PITs on their own lags).

The supervisory data behind the evidence

Under the US Basel III market risk rule, Subpart F, Section 205, paragraphs (c)(1) through (c)(3), firms have reported backtesting data for each subportfolio for each business day since January 2013. The study uses those regulatory reports from January 1, 2013 to December 31, 2015, aggregated to protect the anonymity of individual firms but broken down by product type and region. The sample covers 20 US firms with large trading books, 597 distinct and non-overlapping subportfolios, and a mean of 591 days and a median of 707 days of data per subportfolio.

For every subportfolio and day, each firm sent its supervisor three items: the one-day regulatory VaR at 99%, the one-day clean P&L, and the matching PIT. Clean P&L is a hypothetical P&L, the change in value of the positions held at the previous close, with new trades and fees removed because the VaR model never saw them. Each firm drew its subportfolio boundaries from its own business and risk practice.

Results are grouped in two ways. A single-product subportfolio consists mainly of one product type. The multiple-product grouping counts every subportfolio in which a product appears at all, however small its weight, so one subportfolio can sit in several multiple-product rows.

Counts as printed in Table 7.1 of the original study, which lists only these product types; the share column is derived here.

A histogram of a well specified model’s PITs is flat, with each tenth of the zero to one interval holding about a tenth of the observations. Departures from flatness have a direct reading, because each bar shows how often outcomes landed in that part of the forecast.

• Humped shape. Too many PITs near 0.5 and too few near 0 and 1. The model overstates dispersion, which is conservative misspecification.
• U shape, or spikes at the ends. Too many PITs near 0 and 1. The tails of the forecast are too thin for what the market delivers, which is aggressive misspecification.
• Tilt. More mass on one side than the other points to a biased mean or a skewed P&L the model misses.

What the bank data show

The histogram of total exceedances per subportfolio starts from a benchmark of about 6 breaches, because each subportfolio has roughly 625 daily observations and 1% of 625 is 6.25. Subportfolios with fewer than 6 breaches look reasonable on this count, while those to the right of the benchmark have too many and fail. Ten subportfolios had more than fifty breaches each in the 625 days and were left out of the chart so that the remaining bars stay legible. (The 625-day figure is an approximation; the sample mean is 591 days per subportfolio.)

Pooling all subportfolio PITs into one series and plotting the histogram gives a clear hump in the body together with spikes at both ends. So the models are generally conservative across most of the distribution, yet their tails are thinner than the realized P&L demands. Zooming into the lowest 5% of PITs shows the same split in the loss tail: at the 1% point the models are conservative, but for the most extreme outcomes, below that point, the tail is understated. The firm-wide (top of the house) PITs, pooled the same way, deviate less from flatness in the loss tail, which suggests that losses are modelled more adequately once portfolios are aggregated.

The same story appears when the empirical CDF of pooled PITs is drawn against the 45-degree line of the uniform CDF. Across the body, the curve shows the S-shape of a conservative model. In the left tail of the subportfolio plot, the empirical CDF rises above the diagonal for the very smallest PIT values, which means losses in the far tail are underestimated; the firm-level plot does not confirm that pattern.

Histograms are qualitative. To put numbers on the departure from uniformity, compute the moments of each subportfolio’s PIT series and compare them with the moments of U(0, 1). For the uniform distribution, the mean and median are 0.5, the standard deviation is the square root of 1/12, or 0.2887, skewness is 0, and kurtosis is 1.8. Each deviation has a direct meaning for the model.

• A mean or median away from 0.5 shows a location bias: outcomes sit systematically above or below the forecast centre.
• A standard deviation below 0.2887 shows PITs crowding the middle, so the model overstates P&L variance; above 0.2887, it understates it.
• Kurtosis above 1.8 shows more PIT mass in the centre and at the extremes than a flat distribution allows, the signature of a model that is too wide in the body and too thin in the tails.

Ranges read from the axis end points of the moment histograms in the original study (kurtosis winsorized); the benchmark column is computed here.

At subportfolio level, most standard deviations sit below the uniform value and most kurtosis values sit above it, so VaR models typically overstate variance while understating the kurtosis of P&L. Means stray from 0.5 and skewness appears in both directions. Nine subportfolios had kurtosis above fifteen and were excluded from the plots. The firm-wide PIT moments are far less dispersed, and their kurtosis is much smaller, which repeats the finding that aggregation hides much of the subportfolio misspecification.

Kupiec (1995) tests unconditional coverage: does the observed breach frequency match the tail probability the model claims? If breaches occur more often than 1% for a 99% VaR, the model systematically underestimates risk at that level; if they occur much less often, it overestimates it. Under the null hypothesis the hit series is a sequence of independent Bernoulli trials with success probability p, while the alternative allows a different success probability π:

With T observations of which T₁ are hits, the likelihood under the known probability p, and under a free probability π₁, are

Maximizing the second likelihood gives the observed hit rate, and the test compares the two log-likelihoods:

The statistic is zero when the observed rate equals p and grows as the two drift apart in either direction, so the Kupiec test penalizes too few breaches as well as too many. At a 1% significance level, the chi-square critical value with one degree of freedom is 6.635.

Kupiec results on bank subportfolios

The null is not rejected at the 1% level for 401 subportfolios, 67% of the sample. Of the 196 that fail, 83 have zero exceedances, which signals overly conservative models rather than risky ones. Leaving the zero-exceedance subportfolios aside, 401 of the remaining 514, or 78%, pass. Muni bonds, single-name credit default swaps (CDS) and tranched products show the best pass rates, but these groups are small, which weakens that conclusion. Among the larger groups, linear equities, interest rates, FX and commodities pass at 70% or more as single-product subportfolios and at 64% or more as multiple-product subportfolios (Table 4 and Table 5 set out every product).

The test has a major limitation: it treats the sample as one block and ignores when the breaches occurred. Twelve breaches spread across three years and twelve breaches packed into one turbulent month give the same LR_uc.

Christoffersen (1998), later applied by Berkowitz and O’Brien (2002), shifts attention from how many breaches occurred to whether they arrive in clusters. The hit series is modelled as a first-order Markov chain whose switching probability matrix is

So π₀₁ is the chance of a breach after a quiet day, and π₁₁ the chance of a breach right after a breach. Independence means yesterday’s state does not matter:

Let T_ij count the days in state j that follow a day in state i, and T_i count the days in state i. The likelihood of the Markov model and its maximum-likelihood estimates are

Conditional coverage: both properties at once

An accurate VaR needs unconditional coverage and independence together, so the two can be combined into a conditional coverage test. Its null says that the breach probability is the same after a quiet day and after a breach, and that this common probability equals p:

Both tests condition on the first observation. Asymptotically LR_ind is chi-square with one degree of freedom and LR_cc with two (critical values 6.635 and 9.210 at the 1% level), although the study relies on finite-sample p-values. In practice the test asks whether breaches are as likely after a breach as after a quiet day, and whether both rates differ significantly from 1%.

Results and limitations

For the independence test, 551 subportfolios (92%) are not rejected at the 1% level. Interest rates, FX and commodities pass at 96% to 98% as single-product subportfolios, while single-product linear equities reach 75%; in the multiple-product grouping all four sit between 92% and 98%. For conditional coverage, 422 subportfolios (71%) pass, and the same four product groups again sit among the stronger performers.

Joint tests catch models that violate both properties, but they lose power when only one property fails, because the healthy part dilutes the statistic. The independence test also looks only one day back. A model could show no link between yesterday’s breach and today’s, yet still produce breaches that bunch a week or a month apart, and the Markov test would not notice.

Two further tools target the one-lag blind spot. The Ljung-Box test (1978) checks the autocorrelations of the hit series over several lags jointly, and the duration test of Christoffersen and Pelletier (2004), covered in the next section, looks at the time between breaches. For the Ljung-Box test, with r_k the sample autocorrelation of hits at lag k, the statistic for m lags is

For autocorrelation at a single lag, 387 subportfolios fail to reject the null at the 1% level, reported as 82% of the total. That share reconciles with the 472 subportfolios used in the regression work below, since 387 / 472 = 82%; against all 597 it would be 65%. FX shows the strongest Ljung-Box performance in both groupings.

Regressions of exceedances on lagged information

Berkowitz et al. (2011) extend the idea: if anything known on day t predicts a breach on day t + 1, the model has ignored relevant information. The study regresses the hit indicator on its own lag and, in addition, on lagged VaR or lagged P&L, using two specifications. A linear probability model (LPM) is ordinary regression on the binary outcome, assuming only that errors have zero conditional mean. It is simple and robust, but it has three drawbacks: the error of a binary variable has a Bernoulli structure, fitted values can fall outside the [0, 1] range, and linearity is imposed on the relationship. A logit model solves these problems:

Each regression uses one lagged explanatory variable at a time beside the lagged breach, and a coefficient β that differs significantly from zero signals a model that reacts too slowly to changing market conditions.

Counts from Table 7.6 of the original study; the “at least one lag” column is lag exceedance plus second regressor minus both, and it plus “neither lag” returns N in every row.

In the first logit specification, 54 of 597 regressions show a significant lagged breach, 36 a significant lagged VaR, and 7 both. With the lagged breach alone, 68 subportfolios, about 11%, have autocorrelated breaches.

Source: Iercosan, Shcherbakova, McArthur and Alper, Tables 7.2 to 7.5, regrouped by test. Ljung-Box percentages are computed on the smaller sample used for that test.

n/a: the sample holds no single-product subportfolio of that type.

Christoffersen and Pelletier (2004) showed that the tests above can lack power and proposed using the waiting time between breaches. If breaches cluster, turbulent periods produce many very short gaps and calm periods produce many very long ones. A correctly specified model produces gaps with no memory: the chance of a breach tomorrow does not depend on how long it has been since the last one. The duration between breach i – 1 and breach i is

Because each day’s hit is Bernoulli, the chance that the next breach comes after exactly one day is p, and the chance that it comes after exactly d days requires d – 1 quiet days followed by a breach:

In continuous time this memoryless pattern is the exponential distribution, and the test sets it against a Weibull alternative, which allows the breach probability to rise or fall with the time already elapsed:

A shape parameter b below 1 means a breach is most likely soon after the previous one, which is clustering. Because the null also fixes a = p, the test can flag models whose breaches are dependent as well as models that produce too few breaches.

In the bank sample, 77 subportfolios fail the duration test at the 1% level, 13% of the total, which means the expected time to the next breach is not constant for them.

Exceedance tests give a simple, single-percentile verdict. Testing the PIT series for uniformity and independence uses far more of the information each day provides. Three goodness-of-fit tests compare the empirical CDF of the PITs, A, with the uniform CDF on [0, 1]. In all three, the PITs are sorted from smallest to largest, z_j is the uniform CDF at the j-th sorted PIT (which, for U(0, 1), is the sorted PIT itself), and n is the number of observations.

Kolmogorov-Smirnov: the largest gap

The Kolmogorov-Smirnov (KS) test is nonparametric and measures the greatest vertical distance between the empirical CDF and the reference CDF:

KS is sensitive to differences in the centre of the distribution but weak in the tails, because the empirical and uniform CDFs are both pinned near 0 and near 1 there, so gaps in the tails are naturally small.

Anderson-Darling: weight on the tails

The Anderson-Darling (A-D) test modifies KS by giving more weight to the tails, which adds power when the problem lies there. Since most institutions manage risk with a tail measure such as 99% VaR, tail misspecification is exactly what matters, and A-D is better at finding it. The null is that the PITs are i.i.d. uniform on [0, 1]; the alternative is any other CDF. With the PITs sorted,

Cramer-von Mises: the mean squared gap

The Cramer-von Mises (CvM) test replaces the single largest gap with the average squared gap across the whole distribution, again under the null of i.i.d. uniform values:

All three statistics are small when the PITs fit the uniform well and grow as the fit worsens.

How the bank subportfolios fared

Only 123 subportfolios, about 21%, pass the KS test at the 1% level, which fits the visible departures from uniformity in the PIT histograms. The more powerful A-D test passes just 65, a mere 11%: the stronger the test, the fewer models survive. The CvM test passes 123, or 21%, with single-name CDS, interest rates and FX showing stronger pass rates than other products.

Source: Tables 7.7 to 7.9 of the original study, regrouped by test.

n/a: the sample holds no single-product subportfolio of that type.

Uniformity is half of the i.i.d. U(0, 1) requirement; the PITs must also be independent. PITs live on a bounded interval, which complicates inference, so each one is first mapped to a standard normal variable with the inverse normal CDF, following Berkowitz (2001). Under a correct model the transformed series is i.i.d. standard normal, and independence can be checked by regressing it on its own lag and on lagged P&L.

Counts from Table 7.10 of the original study; the last row is (N minus neither) / N, computed here.

In the simple autoregression, 218 of 597 subportfolios, about 37%, show significant dependence in their PITs, far more than the 11% with autocorrelated breaches in the logit work. A scatter of each PIT against the previous day’s PIT makes the same point visually. Under independence and uniformity the cloud would be evenly spread over the unit square; the bank data show too much mass in the centre and at the extreme corners, which signals both dependence and a lack of uniformity in the joint distribution.

What the full evidence says

Exceedance-based backtesting is an established method, and banks have tuned their models to it: 67% of subportfolios pass the Kupiec test, 78% once the zero-exceedance cases are set aside, and only 11% show autocorrelated breaches in the logit regressions. The distribution-based tests are much harsher. Only 21% pass KS, only 11% pass Anderson-Darling, and 37% show dependence in their PITs. Bank models therefore perform better on exceedance tests than on tests of the whole distribution, plausibly because they are built to be conservative at the regulatory 99% point and carry regulatory add-ons, while the rest of the forecast distribution receives less attention.

For a validator, the lesson is practical. An exceedance test answers one question about one quantile. A PIT-based backtest uses every day’s outcome, checks every percentile, and can reveal a model that is too wide in the body, too thin in the far tail and slow to react, all while it passes the regulatory count.