FRM 7: Principles for Effective Data Aggregation and Risk Reporting
Every risk number a firm produces rests on data that somebody collected, stored and passed along, which is why data belongs on the list of assets a business genuinely owns. Some of it is generated inside the firm: a bank has its own transaction records, and a manufacturer knows what it paid for raw materials. The awkward question about internal data is not whether it exists, but whether anyone kept it in a shape that allows analysis. Other material comes from outside, and public sources cover only part of what risk teams want, which includes histories of inflation rates, movements in the money supply, major interest rates and exchange rates. Alternative data is the label for information gathered by third parties, including data scraped from the web and readings from mobile devices and sensors. Big data means volumes complex enough to defeat ordinary processing techniques, and unstructured data arrives with no predefined data model. Machine learning and artificial intelligence are now the standard tools for turning both into something a risk manager can use.
How the industry acquired a data problem
Financial firms spent decades collecting data department by department, or business activity by business activity, with very little coordination. Separate teams sourced the same feed twice and neither knew about the other. Plenty of data was neglected, and some was destroyed, since migration from one computer system to the next is exactly the moment at which records disappear. Through the 1960s and 1970s the medium was paper cards and computer tapes. Floppy disks and hard disk drives arrived afterwards, and neither could be read by the older generation of systems.
Within the Basel Committee on Banking Supervision, a special committee looked at how banks collected, stored and analysed data. It found problems throughout the industry and published a report on risk data management, concluding that quality was not adequate for aggregating exposures and reporting them at the level of the bank group, or for breaking them down by legal entity and by business line. In January 2013 the BCBS answered with fourteen principles, known ever since by the paper number: BCBS 239.
In the Committee’s own words, risk data aggregation is the work of defining, gathering and processing risk data so that it answers whatever risk reporting requirements the firm has set, which lets the bank judge its performance against the risk tolerance and risk appetite it has chosen. The principles reach risk management data and the models fed by it, and they fall into four groups that the rest of this lesson follows in order.
| Group | Number | Short name |
|---|---|---|
| Governance and infrastructure | 1 | Governance |
| Governance and infrastructure | 2 | Data architecture and IT infrastructure |
| Aggregation capability | 3 | Accuracy and integrity |
| Aggregation capability | 4 | Completeness |
| Aggregation capability | 5 | Timeliness |
| Aggregation capability | 6 | Adaptability |
| Reporting practices | 7 | Accuracy |
| Reporting practices | 8 | Comprehensiveness |
| Reporting practices | 9 | Clarity and usefulness |
| Reporting practices | 10 | Frequency |
| Reporting practices | 11 | Distribution |
| Supervisory expectations | 12 | Review |
| Supervisory expectations | 13 | Remedial actions and supervisory measures |
| Supervisory expectations | 14 | Home/host cooperation |
Source: Basel Committee on Banking Supervision, BCBS 239, January 2013.
A firm that follows the principles closely leaves its risk managers with far less doubt on five fronts: accuracy, integrity, completeness, timeliness and adaptability. Building a real aggregation and reporting capability improves decisions of both kinds, the tactical ones taken this week and the strategic ones taken once a year. Better decisions cut the chance of losses and lift risk-adjusted returns, which is the commercial argument for spending on plumbing that no customer will ever see.
Turning volume into information
Banks have to work out which risk information is worth having, what data can realistically be obtained, and what it will cost. Fast moving big data is hard to refine into something a risk report can carry. What matters is that decision makers trust the underlying data, because where the information reaching them is inaccurate or incomplete, sound risk decisions are not available to them. Advances in data analytics, machine learning among them, collect and convert large volumes of unstructured data into usable information. Organisations then avoid drowning in material they cannot digest, and the volume they do digest becomes a competitive advantage rather than a storage cost.
Model validation and the data behind the model
Rigorous model validation matters here too. Regulatory guidance on model vetting binds model developers in the United States, and the Federal Reserve has published comprehensive guidance for banks on managing model risk. It calls for a “rigorous assessment of data quality” alongside proper documentation, and the developer has to show that the data chosen suits the job and sits consistently with the theory behind the model and the methodology selected.
The rise of the chief data officer
BCBS 239 was a major driver behind the spread of the chief data officer function. That officer normally owns the job of standardising how a firm manages data, and standardisation has travelled well beyond reference data to cover financial products data and accounting data. The Financial products Markup Language shows what such a standard looks like, defining a taxonomy and a structure for derivative products with eXtensible Markup Language standards, and covering the underlying instruments too.
The payoff shows up in what reaches the top of the house. Where departmental applications and methodologies rest on the same standards, the data flowing upward gives a reliable, accurate and manageable picture of the total risk profile. Where they do not, important connections between different dimensions of the business stay hidden.
The Basel Committee wanted firms to have strategies meeting the principles by 2016. That did not happen: banks have struggled with BCBS 239 from the start, no bank at all met the original timeline, and as of mid-2019 most were still finding implementation difficult.
Four reasons compliance slipped
The first is the sheer complexity of the IT reengineering needed to bring a sprawl of legacy systems into line, and the second follows from it, since firms underestimated the effort involved. The third is the dynamic nature of the principles themselves. The fourth is the exponential increase in the use of artificial intelligence techniques on large data sets, which enlarges the surface compliance has to cover. Cost considerations sit underneath all four.
What flawed data actually does
Consider a customer holding credit products in two business lines of one bank, a mortgage loan in the first and a credit card in the second. Without standardised customer identification codes the two records are never joined into one person, and the bank cannot see its own concentration. That is data risk in its plainest form, produced by absent standards rather than by bad judgement.
An operational process that lets flawed data in can also fail in the aggregate rather than at the point of entry. Erroneous and fraudulent mortgage application data, concerning the suitability of the loans, played a part in precipitating the 2007-2008 collapse of the U.S. housing market. Each application arrived one at a time, at an unusually high frequency, and the damage appeared only once the whole book was viewed together.
Data quality and model risk
Financial institutions run their daily operations and analyse their exposures through models, so even a small model error can carry severe consequences. Model risk decomposes into four components. Input risk and estimation risk come first, then valuation risk and hedging risk. Data acquisition bears most directly on the first, because the data creates the statistical estimators of a model’s parameters and errors feed straight into those estimates. The old adage covers it: garbage-in, garbage-out.
A banking group runs its mortgage business and its cards business on separate platforms acquired at different times. Neither platform carries a group-wide customer identifier. The group risk report shows a mortgage book, a card book, and no overlap between them.
Principle 1 asks that a firm keep its aggregation capabilities and reporting practices under strong governance arrangements consistent with the other principles and guidance the Basel Committee has issued. It is placed first deliberately, because nothing further down the list survives an organisation where nobody owns the data. Compliance needs that governance framework together with risk data and IT infrastructure that has been designed well, so Principles 1 and 2 are usually discussed as one block. Independent validation belongs with them, since somebody outside the process has to confirm that the capabilities work as intended and suit the risk profile of that particular firm.
Architecture is not infrastructure
The two words are used precisely here. Infrastructure means the actual components of a system. Architecture means the design of those components and the relationships between them. A system is built on an infrastructure, and that infrastructure has an architecture.
What the board is on the hook for
Most banks were still finding the principles difficult well after the deadline passed, which puts a particular obligation on the board and senior management. They have to understand the limitations preventing effective aggregation and reporting, and then remedy them, since understanding without remediation is not compliance. The board reviews and approves the framework and makes sure the resources to run it are available. Policies are revised where necessary after a major acquisition or a change in strategy, because both events change the data landscape underneath.
Effective and ineffective governance
Governance rated fully or largely compliant comes from policies that set out a clear delineation of roles, of the incentive schemes, and of who is responsible for managing risk data, including dedicated staff whose job is to define what is expected of risk data. Incentives sitting alongside roles is unusual for a technical standard, and it reflects how much of data quality depends on whether anybody is rewarded for it. Non-compliant governance is the opposite: no structured policies or frameworks through which risk data activities are consistently assessed and reported upwards. If risk data is the blood of a financial enterprise, data integration is its circulatory system, and a bank with a limited ability to integrate data will struggle whatever its intentions.
Principle 2 asks a bank to design, build and maintain a data architecture and an IT infrastructure that fully support its aggregation capabilities and reporting practices, not only when conditions are calm but during stress or crisis, while still meeting the other principles. The stress clause is the demanding part, since systems that hold up on a quiet Tuesday are not the ones under examination.
Firms are expected to establish integrated risk data architectures. Roles have to be specified clearly, including who is responsible for adequate controls across the whole lifecycle of the data and for every aspect of the technology infrastructure. No uniform blueprint exists for an infrastructure that complies with BCBS 239, and solutions are specific to each institution. What the best of them share is one condition: every person and every system inside the banking group works from the same data, the same models and the same assumptions.
Why the data is copied rather than read in place
The practical challenge is gathering data from many internal and external sources and delivering it into risk analytics systems. Risk applications do not normally reach into those source systems directly. Instead the information is copied out, extracted, translated and then loaded into a financial data warehouse, and the analytics run against that warehouse. The reason is operational. Analytical processes are computationally heavy, and running them against live systems would degrade the performance and response times the operating business depends on. Data integration is the name for that whole activity, the merging, translating and loading of data drawn from physical sources into a store built on agreed logical and physical models.
Firms are also expected to create information about the characteristics of their data, and data models are the usual form it takes. Four primary types are named, at descending levels of abstraction.
A semantic model settles the agreed meaning of the elements in the model, and it usually travels with a documented understanding of how those elements behave when acting on other elements. Standardisation at this level improves the efficiency and quality of enterprise financial risk management and supports data standards industry wide and globally. A conceptual model confirms human understanding of the system and its objectives, taking a high level view of how groupings of structures, processes and informational elements interact. A logical model expresses the data requirements and properties without tying them to a platform. A physical model translates those requirements into an implementation on a specific hardware and software vendor platform, and can generate the operations, procedures and data loads that create a working database instance.
Banks judged fully or largely compliant on architecture and infrastructure have consolidated their approaches and structures for categorising data and integrated their data taxonomies, with a data dictionary and a single repository or warehouse built for each risk type identified. Non-compliant banks lack the processes and controls that keep risk reference data updated when business activities change, and have no formal route by which poor data quality is escalated to senior management.
Principles 3 and 4 push firms to monitor their data continuously for accuracy and integrity. The bank has to be capable of generating accurate, reliable risk data for reporting under normal conditions and under stress, with aggregation largely automated so that the probability of error falls. Completeness asks that all material risk data be captured across the banking group and be available by legal entity, business line, asset type, region, industry and any other grouping relevant to the risk, so that exposures, concentrations and emerging risks can be identified and reported. Risk data should be reconciled against its sources and granular enough to carry every material risk disclosure. Classifications make that mass of detail presentable to executive management, but drawn too broadly they cause information loss and distort the data.
Timeliness depends on what is being measured
Principle 5 asks for aggregated and up to date risk data produced in a timely manner, while still satisfying accuracy and integrity, completeness and adaptability. How timely is timely varies with the risk area being monitored. Risk information on the trading floor is needed far faster than risk information on a corporate loan. Systems built for trading rooms handle a wide variety of specific and often complex financial instruments, and those positions need evaluating quickly and repeatedly if a trading book is to be managed at all. Such systems apply sophisticated valuation and pricing algorithms, using data structures customised by a vendor or by an in-house team to record the details of instrument contracts. Timeliness is often compromised at the seams, where data must be extracted from several trading systems and mapped into another that can integrate, summarise and report the consolidated position.
Adaptability
Principle 6 asks that a bank generate aggregate risk data for a broad range of on-demand, ad hoc requests: those raised during a stress or crisis, those following a change in internal needs, and supervisory queries. One example of adaptability is taking a hypothetical stress scenario and integrating it with the rest of the portfolio to produce an aggregated enterprise risk measure. Another is absorbing a change in an upcoming regulatory framework, such as an update to the Basel capital rules, and combining it with historical data.
The Basel Committee has described what it observed on both sides of the line, and the contrast teaches more than the principles read alone.
| Area | Fully or largely compliant | Compliance gaps |
|---|---|---|
| Data elements | Individual data elements are certified | Reference data is not standardised |
| Documentation | Data quality is documented | Manual processes relied on with no documentation or policy |
| Assurance | Quality assurance mechanisms are in place | Deficiencies in data quality controls |
| Standards | Quality assessed for each risk type | No minimum standards or thresholds for data quality |
| Ownership | Controls over manual work documented and working | No designated authority with oversight |
| Escalation | Poor quality reaches senior management by a set route | No effective escalation model |
| Checking | Reports reconciled to their sources | Key reports unreconciled, no variance analysis |
| Group reach | Automated sourcing across the group | Risk data not sourced promptly from foreign subsidiaries |
Source: Basel Committee on Banking Supervision, progress report of June 2018 on BCBS 239 adoption.
Two threads run through the right-hand column. One is manual work standing in for automation, which is why Principle 3 asks for aggregation on a largely automated basis. The other is the absent owner, since a control nobody is accountable for is a description rather than a control.
Reporting is where aggregation becomes visible, and banks have ground to cover on this half of BCBS 239. Principle 7 asks that reports be accurate and precise, so that the board and senior management can rely on the aggregated information when taking critical decisions about risk, and that they be reconciled and validated. Where models do the aggregating, those models have to be fully vetted, since results are accurate only to the level of specificity the vetting establishes. Requirements for accuracy and precision are set to match how critical the decisions taken from a report are.
Principle 8 asks for comprehensiveness across every material risk area, with depth and scope matched to the size and complexity of the bank and to what the recipients need. That takes in the Pillar 1 risks of market risk, credit risk and operational risk, and the Pillar 2 risks of business risk, reputation risk and strategic risk.
Principle 9 covers clarity and usefulness. Reports should be clear and concise, easy to understand, and still comprehensive enough to support an informed decision, with a balance struck between risk data, analysis and interpretation, and qualitative explanation. They are meant to be purposeful, tailored to a specific audience such as a trading unit or a lending unit, and a report going to the board of directors should not be hard to interpret at an aggregate level.
Frequency and distribution
Principle 10 leaves the frequency of production and distribution to the board and senior management, set against the needs of recipients, the nature of the risk, the speed at which it can move, and how much the report contributes to sound decisions. Frequency should rise during stress. Some limits are unavoidable, though. Forward looking stochastic cash flow simulations can produce far more output data than they consume as input, and an excess of output undermines the quality checks the bank needs to run. Combining several analyses and model iterations also demands consistent contexts and synchronised scenario parameters, and without that consistency the diversification across scenarios is lost, along with any ability to say what the volatility of the aggregate result is.
Principle 11 covers distribution: reports reach the relevant parties while confidentiality is maintained. An agreed set of distribution lists is the mechanism, and it has to respect that different sections of one report carry different degrees of confidentiality.
A bank sends the same forty page risk pack to its board, to a trading desk head and to the head of corporate lending. It is monthly, accurate, and covers all Pillar 1 and Pillar 2 risks, but it is static, with no way to answer a follow-up question.
The last three principles are addressed to supervisors rather than to banks. Principle 12 asks them to review and evaluate a bank’s compliance with the first eleven principles periodically. Principle 13 asks that they hold, and use, the tools needed to make a bank remedy deficiencies in its aggregation and reporting quickly. Principle 14 asks for cooperation with supervisors in other jurisdictions, which matters for any group operating across borders.
The Risk Data Network releases progress reports at intervals, and supervisors rate how firms are performing. Ratings run on a scale of 1 to 4, where 4 is fully compliant and 1 is non-compliant, with materially non-compliant and largely compliant between them. The table shows average ratings given to 30 banks in 2016 and in 2017.
| 2016 | 2017 | Change | |
|---|---|---|---|
| P1 Governance | 2.83 | 2.90 | 0.07 |
| P2 Architecture | 2.60 | 2.73 | 0.13 |
| P3 Accuracy | 2.60 | 2.60 | 0 |
| P4 Completeness | 2.93 | 2.90 | -0.03 |
| P5 Timeliness | 2.73 | 2.87 | 0.13 |
| P6 Adaptability | 2.90 | 2.90 | 0 |
| P7 Accuracy | 2.77 | 2.73 | -0.03 |
| P8 Comprehensiveness | 3.00 | 3.03 | 0.03 |
| P9 Clarity | 3.10 | 3.03 | -0.07 |
| P10 Frequency | 2.97 | 2.97 | 0 |
| P11 Distribution | 3.37 | 3.33 | -0.03 |
Source: Basel Committee on Banking Supervision, progress report of June 2018 on BCBS 239 adoption; supervisory assessments of 30 banks.
Use the ratings in the table above.
Those small movements make the point: implementation has advanced very little. A 2016 survey by McKinsey & Company and the Institute of International Finance found the same, banks still struggling despite significant investment.
Where the requirement is heading
The principles were written for internal risk reporting, but some supervisors have said that regulatory and stress-testing results would also help them assess a bank. The European Central Bank has more recently stated that financial and regulatory reporting forms part of BCBS 239 compliance, which widens the scope beyond the original paper. Banks have asked for clearer guidelines and regulators have not provided them, holding that assessing compliance is subjective and that the standard applied to each bank is bespoke.