VRM 8: Stress Testing

Scenario design starts with the horizon. One-day and one-week scenarios are used occasionally, but the common range runs from three months to two years, and the criterion is simple: the horizon has to be long enough for the full effect of the scenario to work itself out. Some risks require a great deal longer. A pension plan or an insurance company worried about longevity risk may need stress tests that stretch across several decades, since that is the timescale on which the exposure actually bites.

Repeating the past: proportional changes or actual changes

A historical scenario assumes every relevant variable behaves as it did during a chosen episode. The same distinction that arises in the historical simulation approach to VaR and ES applies here. For variables such as equity prices and exchange rates, it is appropriate to repeat the proportional changes observed in the past. For variables such as interest rates and credit spreads, repeating the actual changes is the appropriate treatment. A stress scenario built from a historical episode therefore mixes the two conventions, applying actual changes to some risk factors and proportional changes to others.

There is no shortage of candidate episodes. The U.S. housing-related recession of 2007-2008, which caused severe problems for many financial institutions, is the obvious one. The collapse in oil prices during the second half of 2014 is another, and so is the flight to quality that followed Russia’s default on its bonds in August 1998. Each carries a different pattern of correlations, which is part of the reason firms keep several of them.

Magnifying a moderate episode

A historical episode that was only moderately adverse can be made more punishing by scaling every risk factor movement by a common multiplier. A six-month loss-making period might be doubled or tripled, and the magnified version can be a serious problem for a firm that shrugged off the original.

The technique assumes that the relationship between risk factor movements is linear, and that assumption does not hold in stressed conditions. Correlations between risk factors tend to rise as conditions deteriorate, so a scenario scaled up uniformly does not reproduce the way variables actually move together when the pressure is severe.

Not every historical scenario spans months. Some are built from what happened to all market risk factors across a single day or a single week, and these are usually sized in standard deviations of the relevant variable, which makes episodes from different eras and different markets comparable.

The extreme reference point in equity markets is a drop of 22.3 standard deviations, recorded by the S&P 500 on October 19, 1987. A firm treating a move that large as beyond the range worth planning for can turn instead to the days around January 8, 1988, when the same index lost 6.8 standard deviations. Two further equity dates carry very large movements: the 9/11 terrorist attacks of September 11, 2001, and the bankruptcy of Lehman Brothers on September 15, 2008. April 10, 1992 supplies the single-day interest rate equivalent, with ten-year bond yields shifting 8.7 standard deviations.

Short-horizon tests of this kind complement stressed VaR and stressed ES rather than duplicating them. The stressed measures take their extreme movements from one chosen period, so whatever was calm in that particular window never enters the calculation. A library of single-day shocks draws the largest movements from many separate stressed episodes, widening coverage without lengthening the horizon.

The second route to a scenario dispenses with history altogether. The risk function simply postulates a large change in one or more key variables and works out what it would do to the firm. The set below is representative of what a bank would run, and the second column is added here to show which part of a bank the shock reaches first rather than being part of the source list.

Source: the variables and the size of each change are the chapter’s illustrative set, reordered here. The third column is editorial.

Why the Greek letters do not carry over

For market risk, a bank’s internal systems already quantify the effect of small changes through the Greek letters, delta, gamma and vega among them. Those sensitivities are local. They describe behaviour in the immediate neighbourhood of current market levels, and a stress scenario moves the variables so far that the approximation stops being usable.

A second limitation has nothing to do with size. The Greek letters isolate one market variable at a time over a short interval, whereas stress testing needs several variables moving together over a much longer period, interactions included, and no set of first-order and second-order sensitivities delivers that. The stress calculation therefore has to be a full revaluation under the scenario.

Regular stress tests, run monthly or on some similar cycle, tell a firm a good deal about the robustness of its financial structure. They do not tell it about the specific thing that might go wrong next. History does not repeat itself exactly, so a programme also needs scenarios shaped around current economic conditions, the particular exposures the institution happens to be carrying, and an up-to-date reading of what could plausibly go wrong. Building those requires managerial judgement, either to write a new scenario or to bend an existing historical one.

The United Kingdom vote on leaving the European Union illustrates the point. For a bank carrying substantial business in the United Kingdom, calling the referendum created a risk with no precedent at all, which is why no historical scenario could have captured it. Ahead of the vote in June 2016, most observers did not expect a decision to leave, and yet the outcome was plainly possible, which is exactly what made it a valid ad hoc stress scenario. Other ad hoc tests might cover a change in government policy on an issue central to the firm, or a Basel regulation obliging the bank to raise capital quickly.

Where the scenarios come from

Adverse views held by professional economists deserve serious attention even when management disagrees with them. During 2005-2006 many economists argued that the U.S. housing market was in a bubble that would eventually burst, and they turned out to be right. A board that rejected the argument still had good reason to run it as a stress scenario, since the cost of testing a view is small and the cost of being wrong about it is not.

Producing answers to what-if questions is not the object of the exercise. Senior management and the board are meant to weigh the findings and decide whether risk mitigation is called for, and involving senior management in building the scenarios makes it far likelier that the results are used. Decision makers tend to fix on the outcome they consider most likely and to dismiss the alternatives as too improbable to act on, a bias psychologists call anchoring, and stress testing exists partly to prevent that.

Boards, senior management and internal economics groups are well placed to do this work, because they combine a reading of markets with a view of world politics and current global uncertainties. One practical device is a brain-storming session run by a committee of senior managers. Research on combining expert opinion suggests that committees of three to five members drawn from different backgrounds perform best. The committee’s most valuable output is not the list of scenarios: it is the set of recommended actions for mitigating the risks that turn out to be unacceptable.

When a scenario comes from a historical episode, most risk factors can be read off what happened during the stressed period, and the effect on the firm follows fairly directly, though judgement is needed on how easily the firm could raise fresh capital or repair its liquidity while the scenario runs. Scenarios built by stressing key variables, and ad hoc scenarios, are harder, because they specify movements in only a few variables and the rest of the picture has to be constructed.

The specified variables are the core variables. Everything else the firm needs in order to value its positions and project its losses is a peripheral variable, and the job of the model is to deduce peripheral behaviour from core behaviour.

One standard approach relates the peripheral variables to the core variables statistically, for instance by linear regression. Which data the relationship is fitted to is what matters. The behaviour of interest belongs to stressed markets rather than calm ones, so past stressed periods make the most informative sample even though they are the smallest.

Credit risk suits this treatment because rating agencies publish long histories. Annual percentage default rates for all rated companies, running from 1981 to 2018, can be related to economic variables such as the GDP growth rate and the unemployment rate to give an overall default rate for any scenario. That rate is scaled up or down for the different loan categories on the books, and a parallel analysis of recovery rates turns default rates into loss rates. For market risk the peripheral variables are those tied to core risk factors such as interest rates and equity prices, and in investment banking profitability tracks equity prices and GDP growth.

A scenario has immediate consequences, and it also has consequences that arrive because other firms react to it. Those second-round consequences are the knock-on effects, and the reason they matter is that the reactions frequently make conditions worse rather than better. Companies defending themselves, particularly other financial institutions, take actions that amplify the original shock.

The clearest illustration is what a scenario built around a possible U.S. housing bubble in 2005-2006 would have looked like at the time. A sensible analyst might have assumed house prices declining by 5% to 10%, and then increased the projected loss on a bank’s mortgage portfolio accordingly. What actually followed was considerably worse, through three channels.

Once some houses were worth less than the mortgage secured on them, owners who could comfortably have kept paying defaulted anyway. In effect they exercised a put option, handing the house to the lender in settlement of the balance outstanding, which worked in states where the lender could seize the property but could not reach the borrower’s other assets. Lenders then sold those properties, adding supply to a falling market, pushing prices lower than they would otherwise have gone and raising the losses on mortgages and on the securities created from them.

A flight to quality ran alongside it. Risky assets of every kind lost their appeal, so equity prices and corporate bond prices fell sharply, and the fall in corporate bond prices meant credit spreads widened. Banks grew suspicious of each other’s creditworthiness and retreated from interbank lending, raising their own funding costs at the worst possible moment.

Ordinary stress testing fixes a scenario and computes the consequence. Reverse stress testing runs the logic backwards, fixing the consequence, failure of the financial institution, and searching for the combinations of circumstances that would produce it. The answer is a description of the firm’s own breaking point.

One practical route uses historical scenarios. The firm takes a series of adverse episodes from the past and asks how much worse each would have to be before the institution failed, perhaps concluding that failure would require a recession three times as severe as the one seen in 2007-2008. Scaling every risk factor movement by the same multiple is an approximation, since the relationships are not linear and correlations climb as conditions deteriorate. A better model builds that tendency in.

Searching across every risk factor for a plausible failure combination is not usually feasible. The workable method fixes a small number of key factors, among them the GDP growth rate, the unemployment rate, movements in equity prices and changes in interest rates, ties every other variable to them through a model, and then searches iteratively across combinations of those factors for the ones ending in failure. The output feeds a stress testing committee, which will throw out some of the combinations as entirely implausible and mark others for closer work.

Alongside the tests a firm designs for itself sit the tests its supervisor designs for it. Regulators in many jurisdictions, the United States, the United Kingdom and the European Union among them, oblige banks and insurance companies to run specified stress tests. The United States example is the Comprehensive Capital Analysis and Review, run by the Federal Reserve across all banks with consolidated assets above USD 50 billion. Participating banks work through four scenarios: baseline, adverse, severely adverse, and one internal scenario of their own.

Neither the baseline nor the severely adverse scenario is a forecast. The severely adverse case is a hypothetical set of events built to test how much punishment a banking organisation can absorb, while the baseline is aligned with what surveys of economic forecasters project on average. Each scenario carries 28 variables covering domestic and international economic activity, among them gross domestic product, the unemployment rate, interest rates and stock market prices.

For 2020, the Federal Reserve set out two hypothetical scenarios, baseline and severely adverse. Its severely adverse case featured a severe worldwide recession, with the U.S. unemployment rate climbing 6.5 percentage points to reach 10 percent, alongside elevated stress across corporate debt markets and in commercial real estate. Banks with large trading operations had to add a global market shock component, weighted that year toward trading book exposures to leveraged loans, and firms with substantial trading or processing operations had to add a counterparty default component. Randal K. Quarles, then Vice Chair for Supervision, described the exercise as a way of seeing how leveraged loans and collateralized loan obligations behave in a recession.

What banks submit and what happens if they fail

A CCAR bank submits a capital plan, documentation justifying the models it has used, and the results of its stress tests. A bank whose capital proves insufficient is likely to be told to raise more, and to face restrictions on the dividends it can pay until it has done so.

Banks holding consolidated assets of USD 10 billion to USD 50 billion fall under the Dodd-Frank Act Stress Test instead. The scenarios in DFAST resemble those in CCAR, but no capital plan is required, because capital management for these banks is handled through a standard set of assumptions.

Setting the scenarios centrally lets supervisors compare banks on identical terms. Regulators are equally clear that they want scenarios developed by the banks themselves, reflecting the vulnerabilities of each institution.

Governance decides how much stress testing a financial institution actually does, whether the assumptions behind its scenarios have been thought through properly, whether senior management gives the results a serious hearing, and whether anything is done about them. Weak governance turns a technically competent exercise into paperwork. The precise structure varies with the legal, regulatory and cultural norms of the country, but a separation of duties between senior management and the board of directors is close to universal.

The board

The board oversees the key strategies. It owns the firm’s risk appetite, meaning the amount and type of risk the organisation is willing to take on in pursuit of its strategic objectives, and it owns the risk culture, meaning the norms of the institution together with the collective attitudes and behaviours of its employees. Since stress testing is one of the principal ways risks get assessed, the board defines how it is to be carried out, which covers the procedures used to create scenarios and the way models and assumptions are applied in evaluating them.

Board members do not perform the tests. They do need enough command of the subject to ask penetrating questions, and they should feel free to draw on their own experience and judgement to demand changes in the assumptions behind a scenario, or to call for scenarios that are entirely new. Where a decision on risk mitigation is in front of them, the board is entitled to ask for further analyses to sit alongside the stress testing output.

Senior management

Senior management ensures that the activities the board has authorised are actually performed, by employees competent to perform them, and reports back to the board periodically on how that is going. It is also accountable for the institution keeping to its own policies and procedures.

Repetition is the standing temptation, because running the same scenarios every cycle is easy and produces tidy trend charts. Senior management is responsible for preventing it: scenarios have to move as the economic environment moves and as new risks appear. Understanding the mechanics better than the board does, senior management is in an even stronger position to challenge key assumptions and models. Severity deserves the same treatment. A scenario built on equity prices falling 20% might be tightened so that they fall 30% instead, as volatilities rise.

Both bodies share responsibility for coverage across all business lines and exposures, for aggregation into an enterprise-wide picture, and for judging whether the results call for more capital or better liquidity. Timing makes that judgement urgent: once an adverse scenario is under way, the room to manage capital and liquidity is far narrower.

A financial institution needs written policies and procedures for stress testing, and it needs to follow them. Their function is consistency: parts of a large organisation will otherwise approach the exercise differently, and results produced differently cannot be aggregated. The table lists what those policies should contain and what goes wrong when an item is missing.

Source: the policy requirements are the chapter’s list, reordered. The consequence column is editorial.

Validation and independent review

Reviews have to be run by people independent of those who conducted the stress test, so that the board receives unbiased assurance that the work matches the firm’s policies and procedures. A review should reach the qualitative and judgemental parts of the exercise, confirm that the tests rest on sound theory, check that limitations and uncertainties have been acknowledged, and monitor results on an ongoing basis. Models bought from vendors face the same scrutiny as models built in house, and every model a firm uses is independently reviewed whether or not it belongs to the stress testing framework.

Validating a stress testing model is harder than validating most models, because rare events leave so little data behind. A one-day VaR model with 99-percent confidence can be checked by counting the days on which losses would have exceeded it, and roughly 1% is the expected figure. Nothing equivalent exists for a stress test, whose output describes an extreme that may never have occurred. Reviewers therefore concentrate on conceptual soundness and check that the model reflects stressed market behaviour, where correlations increase and recovery rates decline. Where the right model is unclear, comparing several is more informative than defending one, and a range of possible losses often serves a decision maker better than a single estimate.

Internal audit occupies a different position again. It does not conduct the stress testing, and it is not asked to. Its job is to confirm that suitably qualified employees run the tests, that documentation is adequate, and that models and procedures have been through independent validation.

The distinctive feature of the internal audit role is its scope. Rather than examining one model or one scenario in detail, internal audit assesses the practices used across the entire financial institution and asks whether they are consistent with each other. That vantage point often reveals ways in which governance, controls and allocation of responsibilities could be improved, and internal audit can then advise senior management and the board on the changes it thinks are warranted.

Basel Committee publications have pressed the importance of stress testing repeatedly. Market risk capital based on a bank’s internal VaR and ES models has to come with stress testing the Committee calls “rigorous and comprehensive”, and any bank setting credit risk capital through the Basel II internal ratings-based approach must stress test its assumptions.

In May 2009 the Committee published stress testing principles for banks and their supervisors. The 2007-2008 crisis shaped them heavily, and they treat stress testing as the way a bank establishes how much capital it needs to absorb losses from large shocks. The Committee lists several roles for the technique: forward-looking assessments of risk, a way past the limitations of models and of historical data, support for communication inside the firm and outside it, an input to capital and liquidity planning, a guide to setting risk tolerance, and a basis for risk mitigation or contingency plans across a range of stressed conditions.

Stress testing matters most, on the Committee’s reading, after a long stretch of benign conditions. The quiet years before the crisis lulled banks into a false sense of confidence, and calm of that kind breeds complacency and the underpricing of risk.

What went wrong before the crisis

Reviewing pre-crisis practice, the Committee reached four broad conclusions. The first concerns involvement: top management and board members should help set objectives, define scenarios, discuss results and decide on actions. Banks that came through the crisis well had senior management actively engaged, and fed the output into strategic decisions. Elsewhere the exercise was mechanical, disconnected from decisions, and run inside business lines with no attention to interactions and no enterprise-wide total.

The second concerns methodology. At some banks the methods could not aggregate exposures across different parts of the bank, and experts in different areas did not cooperate on a firm-wide view, so the optimism of mortgage-backed securities traders was never tempered by what retail lenders were seeing. The methods also assumed average historical relationships between risk factors would hold, and they omitted knock-on effects.

The third concerns severity. Scenarios were too mild and too short in duration, correlations between different risk types, products and markets were underestimated, and reliance on historical scenarios crowded out the risks created by new products and newly taken positions. The fourth concerns coverage. Particular risks went untreated in sufficient detail, among them structured products, products awaiting securitization, imperfect hedging and counterparty credit risk. Liquidity effects under a stressed scenario were underestimated, and the crisis produced systemic risks as banks hoarded liquidity and refused loans they would have granted in normal conditions.

The revised principles

Having watched practice develop after the crisis, the Committee issued a consultative document with a revised set of principles in 2017, intended for national authorities designing stress testing rules, guidance or principles of their own.

Source: condensed from the December 2017 consultative document Stress Testing Principles, issued by the Basel Committee on Banking Supervision. The wording is a summary rather than the text of the principles themselves.