Overview
Join our host Scott Sobolewski as he interviews Roland Stamm, Partner within our Quantitative Services division on the important topic of model validation both for ISDA SIMM and complex internal models.
Roland stresses the importance of model governance and how it is crucial for firms to be able to prove to regulators that their model is fit for purpose.
Packed with sound advice and insights, this podcast is a must for firms that are embarking on backtesting and benchmarking projects.
Listen to the podcast
Hello and welcome to Ahead of the Curve, the podcast from Acadia,where we dig deep into relevant topics around the world of derivatives,risk management, and collateral. My name is Scott Sobolewski,co-head of Acadia's Quant Services division,and today I'm joined by Doctor Roland Stamm,a partner in Acadia's Expert services group and oneof the founding partners and original authorsof the open source risk engine for listeners from previous podcasts.The open source risk engine powers the majorityof what Acadia offers for our hosted risk services.Roland also brings over 25 years of experienceworking with clients primarily on a consulting basis,clients in Europe, the UK, and now more and more in Asia, we see.Supporting clients who come to Acadia, both for the unclear margin rulesand various model development and model validation projectsaround the edges of EMR, as well as bespoke internal modeldevelopment and model validation requirements. Welcome, Roland.Thanks, Scott. Great to be here. -I figured we'd start firstabout your career journey to Acadia, just to provide folksa bit more information about your backgroundand your credentials here. Today, we'll eventually be talkingabout SIM model validation. However, given you have, I'd saymany more years by now, historically supporting, I'd saymore complex internal model development validation initiativesat large investment banks. Maybe you could just give the listenersa flavor of some of that experience. -My original trade is mathematics.In 1999, I started in the banking industry,in a relatively small bank, which went through a lot of different phases.It was split up, it was bought. It was going into defaultand had to be saved by the German governmentwas split up again into multiple parts. There was a very interesting journeyover 14 years, which gave me a lot of experience.I started in IT, basically, programming reportsand pricing functionality for the front office,then moving into the risk space, doing mostly pricing first,but then also risk modeling, and becoming responsiblefor that in the organization. After that very interestingand long journey, I decided to join some old colleagues of minewho had founded this very interesting consulting businesscalled Quaternion. In 2013, I joined them,and the rest is history. We started as a relativelysmall consulting firm and was then bought by Acadia, in 2021?Yes. -After we had a very successfulpartnership on the services that you already mentioned.Then last year, we became a part of the family.Yes. We found a very happy home here in the L6 Post-trade solutions group.Those of you who are coming to this podcastprobably know Acadia as the central industry utilityfor initial margin reconciliation, primarily within the non-cleared universe.The industry model that's used to calculatethat initial margin is the ISDA standard initial margin model,or the SIM for short. Compared to someof the more complex model development or model implementationsthat you've been a part of in your career, large potentialfuture exposure implementations. Very large-scale Monte Carlo risk factorevolution implementations of XVA within the front office and risk.Maybe Roland, you could just tell us a bit about why the SIM model is uniquecompared to some of those potentially more complex models,and what clients need to be aware of when they're lookingto implement or validate ISDA SIM locally. -ISDA SIM is unique inthat it isa very prescriptive model if you want. I wouldn't even call ita model as such, really, because-- -A big formula.A big formula. Exactly. That's what it is, right?It is coming,or it is similar to regulation FRTBSA,which is short for the fundamental review of the trading book standard approach,which is supposed to be the standard approachfor calculating capital requirements for market risk.It's actually a bit simpler than that approach,but very similar in terms of the input factors that are required.On the one hand, it takes away a lot of uncertainty,but you still have remaining uncertainty in that.You have to obviously still calculate the input,which is NPVs, and the changes as the market data changes.Of course, that depends on how you use market dataand how you calculate the NPV. That is something that is more divergingthe more complex the product becomes. If you have a plain vanilla swapor an FX forward, something like that. Probably most people would agreewith what comes out at least with the same market data.Whereas when you have, I don't know, a tariff or other very exotic products,people will start to disagree, even if they use the same market data,right? Because they use different model approaches.That is always underlying these calculations.In that sense, it's a very simple formula, but with remaining uncertainty.That's I think where a handy industry standard in even calculatingthose sensitivities is needed and a good thing to have.It does make the calculation and reconciliationof the margin amount easier from our client's perspective.What they don't often realize up front, if they don't have much experiencewith model validations or the pursuit of regulatory model approvals.At least for SIM, it's really, as you mentioned,much more about validation of the underlying pricing modelsproducing those risk sensitivities. One of the big points of communicationthat we'd like to come across in this podcastis the potential burden on clients to validate SIM.It's not just one model that you're validating.It depends very heavily on the diversity and the complexityof the underlying product types or instrument typeswithin your eligible portfolios, and depending on those implementationsand the product coverage, validating your SIM implementationor your model governance framework around SIM can require a validationof potentially dozens of individual models producing those risk sensitivities.Not to mention the SIM-specific validation requirements around back testingand benchmarking. However, maybe before we get into thosebased on, as you mentioned, some of your pre-2008pre-crisis experience and your post-crisis experience,maybe you could just speak to the evolutionfrom your standpoint of model validation and model governance standardsover that time frame over your career and particularly from the perspectiveof a regulated entity like a bank or a systemically important institution.Yes. I think the turning point in terms of model risk and awarenessthat model risk management is needed was really the financial crisis of 2008,2009, even though before we also saw historically models breaking down.The best example, I think, is 1987. I think it was Black Mondayor Tuesday, I can't remember, where basically the market realizedthat Black Scholes with the standard volatility doesn't workand that a smile is required, that is where the smile emerged, kind of.That wasn't really until 2008/2009, when regulators realizedsomething was really amiss. I mean, there wasone particular trade type, the CDO and CDO squared,these kinds of things, which broke down entirelybefore everybody in the market had been using the same model.A single pricing parameter, basically the correlation to price them.They realized, okay, this is completely wrong.Just saying it's industry standard was not enough anymore.Everybody realized we need something where people are really owning the model.Everybody who uses it has to own it. They have to show that they understand itand that they understand the breaking points as well.They can't just rely on vendor documentation.I think pre-2008, even large bank model validation functions,the model owner could probably get away with just passing along the documentationas given to them by the vendor. Now, as you mentioned,the regulatory pressures on demonstrating and ownershipand a regurgitation or translation of any sort of even vendor modelinto the client's own understanding that that's a critical pieceof the validation lifecycle. -Yes. One of the most prominent examplesin the US for that kind of new awareness by the regulatoris the SRA 11 seven standard, which sets a pretty high barto pass for model validation, where it saysyou not only need to understand the model, you haveto do benchmarking, you have to do alternative model implementationsto show that you are in a certain range. You have to explainwhy you are choosing this model and not another one, and so on and so on.Not only benchmarking model outputs, but benchmarkingand having demonstrated quality controls around inputs like portfolio data,and even more so, the market data, as you mentioned.Exactly. You have to actually describe the entire processingfrom input data to output data, how the data is manipulatedto get to the result, et cetera. Yes. -That SR 11 seven documentationor regulatory guidance originated in the US,but we do see across our global client basethat continues to be the gold standard and is being referredto by other regulators globally as the standardthat they expect when they go and do any sortof regulatory approvals. For the majority of our clients,Acadia's hosted risk calculation service, the fifth generation service,and our backtesting and benchmarking service for clientswho have a regulatory requirement to have their initial margin modelapproved before use. We've had by now had over 100 clientsvalidate the Acadia-hosted offering. Based on everything we just talked aboutfor the last 10 minutes or so, validate the individual pricing modelsthat Acadia is offering and the market datathat Acadia is offering. However, in their own languageand their own internal documentation, and making sure that the input and outputbenchmarking the qualitative methodologies that Acadia has chosen fitnot only their model owner's purpose, but also passan independent model validation from the client's perspective.Being part of those dozens and by nowover 100 model validations, we know the types of questionsand the types of issuesor segments of relevant model owner or model validationdocumentation that many regulators in advanced jurisdictionswould expect, particularly as it relates to the Anderson modeland some of the underlying pricing models that are producingthose risk sensitivities. -A new example of thatis the latest version of the so-called RTS regulatory technical standardsaround initial margin validation. The EBA, the European Banking Authority,has issued a draft for how to validateyour initial margin model. It has recognized, okay,there are many financial market participants who are exchanginginformation, how are they doing it? What is the model behind this?In the vein of SR117or similar regulations on other risk models,they want the users to own the models, to understand them,to inform their regulator how they do it, and explain the details.An important part of that is backtesting. Even if you are only a small user,they now distinguish between smaller and larger participants.Phases one through four are in that large group,and then phases five and six are generally basically smaller group.That's the split exactly. The smaller group will have fewer hoopsto jump through. However, there's still a backtestingeffort that needs to be made in order to prove that you are using a modelthat is fit for its purpose, for your particular portfolio.One tangible example, I believe that larger groupis subject to two types of back testing at quarterly static back testingthat is based on hypothetical or risk theoretical panelsgenerated historically, in addition to dynamic back testingwhere SIM is being compared against realizedactual panels, whereas for that smaller group, they're only subject to oneof those two types of back testing, the dynamic back testing, I believe.It does limit the model governance burden to some extent,though, especially for banks that have their own internal modelgovernance functions or certain other regulated entities,they'll still need to produce the documentation,at least initially, that codifies how the modeloperates, what sort of inputs are required, the processflows, the benchmarking, the quantitative benchmarking,which in our experience is where the majority of the effort lies.Even I think what a lot of our clients don't realize initially,especially if they don't have a lot of backgroundwith dealing with regulators for model approvals,is that they need to validate or receive approvalfor the ISDA SIM model. It's actually the ISDA SIM framework.It's not just one model, it's not just one set of documentation.Even for a probably the simplest portfolio we've seen,where it's one or two product types, let's say it'sone vanilla interest rate swap and European FX options, right?Two product types in total. That requires up to nineor 10 distinct sets of documentation. From the model owner standpoint,one document each for the pricing models for those two products,one model owner document for the ISDA SIM model itself,and then generally another model owner documentfor the benchmark VaR model that's being used to compareagainst SIM on a daily or quarterly basis. -All the backtesting framework.Yes. Generally, that can include the penile generation methodologiesused in backtesting. That's right, they're justfrom the model owner's perspective, even for clientswho are using Acadia's outsourced risk generation and backtesting services,they'll still need to produce those four documents themselves.Additionally, the same set of documents, but again authoredby an independent model validation group, so written morefrom an independent perspective that's based on the dataconsumed by those model submission documents.That's eight documents right there. Generally, there's a third line,an internal audit function or an internal controls functionthat assesses that end-to-end framework, usually on an annual basis,sometimes less frequently for smaller institutions,that document itself needs to be codified and reviewed by regulatorsand more from a retrospective standpoint. Then lastly, some of our clientswho are not banks, maybe they're larger buy-side firms.In the US, we have this concept of swap dealersor security-based swap dealers. It's a regulatory distinctionfor non-banks that are still systemically importantin the derivatives market. For some of those clients,they're standing up a model governance frameworkor a model risk management framework for the very first timebecause of their regulatory approvals or their regulatory capital requirementsin the US. That stuff only went live in 2021 or 2022.They're standing up an entire model governance frameworkfor the very first time. You need to have a governance documentthat outlines all of the requirements that the validation teamwould work towards. Model scoring criteria.Model tiering criteria, basically an entire frameworkthat decides how these models should be validated internally.Again, that's unique to each individual institution.Hundreds of pages of documentation, even for very simple vanilla portfolios.You can start to see how the documentation burdenand the resource burden start to increase exponentiallyas the number of different product types in scope for you and Omar grows.We see clients sometimes with up to 50 different product typesacross all six asset classes. Those generally tend to bethe larger investment banks. You can start to understand very quicklywhy they have armies of hundreds of quants on standbyto produce this documentation, not only for UMR or SIM purposes,but more broadly as well. Roland, in my experience in the US,the question of regulatory approval for SIM has generally been answeredfor the majority of phase five and phase six clients.We've supported clients on a consulting basisthrough our expert services group in those model validationsand regulatory approvals. The two big pieces of advicethat I would give to clients who have not yet gone through that journeywith their local regulator in places like Europe or the UKare to first be proactive, both in terms of timeline planning.Even simpler implementations or simpler regulatory approval timelinescan take up to six to nine months, and more complex validation lifecyclescan be over a year, especially when setting up a lotof this documentation for the first time. Definitely try not to waittill the last minute for any looming regulatory deadlines.Regulators themselves are resource-constrainedlike any other institution. The second big piece of advicewould be to be proactive in reaching out to your local regulatorto declare your intentions for use of ISDA SIMor an initial margin model. They're generally not only queuesor lines to get in. The sooner you can get in that regulatory queue, the better,but even more importantly, most regulators,in response to that outreach, will send in the US.They would call them a first day letter, but essentially a setof formal regulatory requirements from your local regulator,that is, essentially your checklist or your shopping listto plan out your pursuit of that regulatory approval for ISDA SIM.I think we're all very curious, particularly in Europe,to know how the future will evolve for the IBA requirements in particular,and what some of those first day letters are startingto look like for maybe smaller entities of some larger institutionswho are getting this outreach now very recently.We have seen some interest coming from smallish clientswho have been approached by the EBA alreadyand have been told that they should do a validation by,let's say, the end of next year. As you say, Scott, I thinkit's very important that you take this seriouslyand that you work proactively on getting those targetsonline, right?Don't hesitate. We're more than happy to help.As you say, we have done this exercise a hundred times.We know what the regulators are looking for, what they're asking for.If any of our listeners need help, we're more than happy to do that.Not only the decades of experience from consultants like Rolandand similar partners in other global regions here in Acadia.However, you don't need to be a user or client of Acadia'sfrom the hosted commercial services that I mentioned earlier.Even if you're using a competing vendor or a completely internal bespokeimplementation to produce the risk sensitivities feeding SIM.We found Acadia can bring lots of synergies to a validation project,particularly on the benchmarking side, where we know we havean industry-leading service that has already been validatedby over 100 clients. We can very quickly and easilydeploy the outputs of our service from our industry standard modelsas a quantitative benchmark against any sort of local deploymentor alternative vendor deployment, producing those sensitivities.At this point, I'd like to point out that we are also sitting on a heapof very good market data because part of our servicesis the SIM back testing service, which, as we said before,is going to be an integral part of a validation exercise anyway.Actually getting clean, good historical market datais one of the biggest challenges in these validation exercises.I just want to point out that we actually have that.I will. Thanks for the plug, Roland. I will point out that we spent 25 minutestalking about exactly that on the market data frontin exactly the previous ahead of the curve episode here.I'm assuming, depending on when that comes out,either the previous one or the one following this episode,so definitely more information to learn about the market data question,which is a very big question for many of our clients.However, a question that we've solved for at Acadia,at least for the hosted services clients. With that, Roland,is there anything else you'd like to ask? No. I think our big messageis just to be quite proactive in your outreach to your local regulatoror your outreach to Acadia for potential assistance,and your internal resource planning. That concludes today's episodeof Ahead of the Curve. For more information about Acadia servicesor anything we spoke about today, we'll have more information at Acadia.Or you can reach out to our general email at info@arcadia.inc.Thank you guys very much for listening and joining us today.Thank you, Roland, for joining me.Thank you for having me.
