What is model risk, and how do banks validate their risk models to avoid catastrophic failures?

Question

AcadiFi · Accepted Answer

Model risk arises when a model used for decision-making produces incorrect or misleading outputs. After the 2008 crisis exposed massive model failures — CDO pricing models, VaR models, stress testing models — regulators mandated formal model risk management frameworks.

**Sources of Model Risk:**

**1. Specification Error (Wrong Model)**
- Using a normal distribution when returns are fat-tailed
- Assuming constant correlations when they spike in crises
- Example: Fairview Capital's VaR model assumed log-normal returns for structured credit. During the crisis, correlation between mortgage tranches jumped from 0.3 to 0.9, and the model underestimated losses by $420 million.

**2. Implementation Error (Right Model, Wrong Code)**
- Programming bugs, data feed errors, incorrect parameter mapping
- Example: A quant at Osprey Trading implements a Black-Scholes model but accidentally uses calendar days instead of trading days for the time parameter, inflating option values by ~15%.

**3. Calibration Error (Right Model, Wrong Parameters)**
- Using stale parameters, insufficient historical data, or survivorship-biased samples
- Example: Calibrating a GARCH model with only 2019-2021 data (including the COVID crash) produces volatility estimates biased upward for normal markets.

**4. Application Error (Right Model, Wrong Use)**
- Using a model outside its intended scope or for decisions it wasn't designed to support
- Example: Using a short-term VaR model calibrated for liquid equities to estimate risk on illiquid private credit positions.

**The Model Validation Framework (SR 11-7):**

The Federal Reserve's SR 11-7 guidance establishes three lines of defense:

**First Line:** Model developers (quants who build and calibrate)
**Second Line:** Independent model validation team (challenges assumptions and tests independently)
**Third Line:** Internal audit (reviews the overall framework)

**Key Validation Activities:**

**A. Conceptual Soundness Review**
- Is the mathematical framework appropriate for the product/risk being modeled?
- Are the assumptions reasonable and well-documented?
- Review published literature — is there academic support for the approach?

**B. Backtesting**
Compare model predictions to actual outcomes:

**VaR Backtesting Example:**
Overton Risk Analytics runs a 99% 1-day VaR model over 250 trading days. At 99% confidence, you'd expect ~2.5 exceptions (days where losses exceed VaR).

| Zone | Exceptions | Conclusion |
|---|---|---|
| Green | 0-4 | Model is acceptable |
| Yellow | 5-9 | Investigate further |
| Red | 10+ | Model is likely flawed, capital surcharge applied |

If Overton observes 12 exceptions in 250 days, the model is in the red zone — it's systematically underestimating risk.

**C. Benchmarking**
- Compare model output against alternative models or industry standards
- If your proprietary model says a CDO tranche is worth $82M but Bloomberg's model says $71M, the $11M gap needs explanation

**D. Sensitivity Analysis**
- Stress individual inputs to understand model behavior at extremes
- How does VaR change if you double the assumed correlation? If volatility triples?
- Identify which inputs the model is most sensitive to

**E. Outcomes Analysis**
- Track model performance over time (P&L attribution, actual vs. predicted defaults)
- Deteriorating performance triggers model review or retirement

**Model Risk Mitigation:**
1. **Model inventory:** Maintain a complete registry of all models with risk ratings (Tier 1/2/3)
2. **Annual review cycle:** High-risk models validated annually, lower-risk on 2-3 year cycles
3. **Model limits:** Set usage boundaries — "this model is valid for tenors up to 5 years and notional up to $500M"
4. **Champion-challenger:** Run the production model alongside an alternative to continuously benchmark

**Exam Tip:** FRM Part II frequently tests the SR 11-7 framework, the Basel traffic light approach to VaR backtesting, and the ability to identify which source of model risk a scenario describes. Know the three lines of defense cold.

Practice model risk scenarios in our FRM Part II question bank.

What is model risk, and how do banks validate their risk models to avoid catastrophic failures?

Master Part II with our FRM Course

Related Questions

Practice Questions