We are included in the Jan 2019 edition of Intelligent Risk on the topic of low probability default.
Quant Foundry –Low Probability Default Modeling Using A Tree Approach
A Model Driven By Market Demand and Fragmented Data
The drivers underlying development of the methodology described in this article arechallenges posed by regulators, and the desire of asset and treasury managers to improve their portfolio investment decisions for low probability defaults (LPD) such as sovereigns, through improved forecastingmethods.
LPD by definition happens infrequently, and the available data is restricted, as there are no more than two hundred sovereignissuers. Traditional statistical techniques that link disparate probability distributions are hampered by fragmented market data, with quality concentrated in developed country issuers.
Countries fail on their external obligations for many reasons, but there is a pattern of behavior that an analyst would look for in the anticipation that history does repeat itself. Examples include material expansion of external debt, ambitious infrastructure programs, autocratic or corrupt government and low GDP per head of population. Market data such as yield curves provide a consensus of relative risk-reward from investors based on capital flows, but such data does not capture the whole and tends to understate the credit risk of sovereigns.
The expertise incorporated by the methodology designers Chris Cormack and David Kelly together combines a combination of industry credit risk knowledge, traditional analytical methods and deep understanding of Neural Network tools, to leverage off a wider selection of empirical data. The approach taken is to break down the thought process of a credit analyst who uses both market and non-market data indicators to replicate their thought process and replicate it using a model that provides a powerful, unbiased and potentially superior default predictor.
How The Model Works
The development of the model starts with acquiring a wide set of data across several countries that are then refined to a minimum “Feature set” that becomes the key influencer driving the model. A Neural Network algorithm is deployed on the historical data of the Feature Set, as well as the prevailing rating, to understand the complex interplay of conditions that presage a rating migration or outright default.
A common pitfall in developing all machine learning solutions is to assume that the use of an ever-larger Feature Set means higher accuracy and better results. For sovereign debt, not all the countries have reliable historical data for all members of the Feature Set. Increasing the Feature Set comes with the cost of dropping some countries for which accurate data is a challenge and increases the computation time. Considerable amount of judgement is therefore applied to overcome variances in the quality of the historical dataset and find an elegant compromise
The appropriate learning algorithm – an enhanced gradient boosted (GB) algorithm in this case – can now be deployed to minimize the difference between the actual rating and the predicted rating. GB is an ensemble technique that combines multiple weak learning algorithms that individually are based on single decision trees. The key advantage of a GB algorithm lies in the fact that it learns from its errors and continues to refine its predictions until no further improvements can be made.
The model can now provide a probability estimate for how each issuer will migrate to each rating from AAA to D during each quarter in the coming year and weighs each member of the Feature Set by its ability to influence the outcome. Stability is checked by creating a forward simulation of the Feature Set and checking that the expected ratings don’t flip on small changes in the data.
One of the biggest challenges is model explanation and interpretability, so having a tight Feature Set is critical. WithNeural Networks it’stricky to understand how each member of the Feature Set influences theoutcome. Given thatGB grows iteratively like a tree, greater influence can be granted purely based on whether a member is present at the start of the simulation or the “root” rather than at the end or “the leaves”. The solution is to deploy a visualization algorithm that looks at the average difference in predictions by un-blinding the test data set – i.e. by mixing up which members are in the root and which are in the leaves. This approach provides a holistic measurement of how each member of the Feature Set influences therating.
The confusion matrix below highlights the high concentration in the diagonal that indicates overall model accuracy and the enhanced predictive power around rating transitions.
The graph below shows the strength of the predictability of the model for highly-rated, and thus low-probability default countries and shows impressive performance. For the next phase of this model development, the plan is to capture the dynamics of cross-over movements, in particular Southern European members of the Euro that do not control their domestic currency, as well as using non-empirical data such as satellite imagery, that captures an assessment of theimpact of prolonged drought.
Conclusion
The use of a blend of credit knowledge, traditional modelling approaches and Machine Learning, together with a robust test framework compatible with SR11-7 guidelines, has improved the predictability of LPD where the prevailing data continues to be a challenge. The results show a step improvement in reducing bias in the credit analyst process and, given its level of automation, reduces overall cost of ownership.
Authors
Dr Chris Cormack is co-founder and Managing Director of Quant Foundry. Chris was former Head of Market Risk with a good understanding of the issues and requirements of Market risk measurement. Experienced Market Risk methodology across all asset classes, advanced Operational Risk methodology implementation. Counterparty Credit Risk Time Series modelling specialising in Equity and Rates. Design and implementation of Novel Time Series models for CCR and Market Risk. Chris has lead quant teams for large methodology integrations. Chris was a lecturer in physics at Queen Mary University and holds a PhD in particle physics from Liverpool and a master’s in mathematical finance from Oxford.
David Kelly is co-founder and Managing Director of Quant Foundry. David has held a number of senior leadership and technical SME roles in the front office, market risk, model validation and counterparty risk. David has been instrumental in the application of Risk Architecture designs that lead to pragmatic delivery of advanced modelling solutions and system changes to requirements under regulatory directives including Basel 2.5, CVA, Stress Testing, IFRS9 and FRTB as well as developing several model risk governance frameworks and TRIM remediation programs. David read Pure Mathematics at Bristol and Part III at Cambridge.