经济资本模型验证方法PPT课件

1、Introduction, Background and MotivationThe validation of EC models is at a very preliminary stageInternal capital adequacy assessment process (ICAAP): not a model for EC, but an overall framework for assessing if EC is appropriate EC may be a quantitative component of ICAAP, but it is not required o

2、f all banks by supervisors (only the largest) Validation of the quantitative EC component of ICAAP, if there is one, is a component of ICAAP EC models can be complex, having many components, and it may not be immediately obvious that a such model works satisfactorily Models may embody assumptions ab

3、out relationships amongst or behavior of variables that may not always hold (e.g., stress) Validation can provide a degree of confidence that assumptions are appropriate, increasing the confidence of users in the model outputs Additionally, validation can be also useful in identifying the limitation

4、s of EC models (i.e., where embedded assumptions do not fit reality)第1页/共36页Introduction, Background and Motivation (continued) There exists a wide range of validation techniques, each providing evidence regarding only some of the desirable properties of a model Such techniques are powerful in some

5、areas (risk sensitivity) but not in others (accuracy of absolute EC or quantile estimator) A range of validation techniques can provide more substantial evidence for or against the performance of the model Particularly in an environment of good controls and governance There appears to be scope for t

6、he industry to improve the validation practices that shed light on the overall calibration of models Particularly in cases where assessment of overall capital is an important application of the model第2页/共36页Fitness for Purpose of Economic Capital Models In some cases the term validation is used excl

7、usively to refer to statistical ex post validation (e.g., backtesting of a VaR) In other cases it is seen as a broader but still quantitative process that also incorporates evidence from the model development stage Herein, “validation” is meant broadly, meaning all the processes that provide evidenc

8、e-based assessment of a models fitness for purpose This assessment might extend to the management and systems environment within which the model is operated It is advisable that validation processes are designed alongside development of the models, rather than chronologically This interpretation of

9、validation is consistent with the Basel Committee (2005) in relation to the Basel II Framework However, that was phrased in terms of the IRB parameters & developed in the context of assessment of risk estimates for use in minimum capital requirements Validation of EC differs to an IRB model as t

10、he output is a distribution rather than a single predicted forecast against which actual outcomes may be compared 第3页/共36页Fitness for Purpose of EC Models (continued) EC are conceptually similar to VaR models, but several differences force validation methods to differ in practice from those used in

11、VaR Long time horizon, high confidence levels, and the scarcity of data Full internal EC models are not used for Pillar 1 minimum capital requirements, so fitness for purpose needs to cover a range of uses Most of which and perhaps all these uses are internal to the firm in question Note that EC mod

12、els and regulatory capital serve different objectives & may reasonably differ in some details of implementation BCBSs Validation Principle 1 refers to predictive ability of credit rating systems, an emphasis on performance of model forecasts The natural evolution of this principle for EC is that

13、 validation is concerned with the predictive properties of those models I.e., embody forward-looking estimates of risk & their validation involves assessing those estimates, so this related principle remains appropriate Broadly interpreted validation processes set out herein in different ways al

14、l provide insight into the predictive ability of EC model第4页/共36页Providing Confidence Regarding EC Model Assumptions Properties of an EC model that can be assessed using powerful tools, and hence that are capable of robust assessment, include: Integrity of model implementation Degree to which ground

15、ed in historical experience Sensitivity to risk & to the external environment Good marginal properties Rank ordering & relative quantification Properties of an EC model for which only weaker validation processes are available include: Conceptual soundness & validity of assumptions Degree

16、 to which model is forward-looking Absolute risk quantification & predictive accuracy of risk estimate It is important to stress the power of individual tests & acknowledge that views as to strength and weakness are likely to differ第5页/共36页Providing Confidence Regarding EC Model Assumptions

17、(cont.) There is great difficulty in validating conceptual soundness of an EC model due to many untestable or hard-to-test assumptions made: Family of statistical distributions for risk factors Economic processes driving default or loss (e.g., observable vs. latent) Dependency structure among risks

18、or losses (e.g., copulae) Behavior of management or economic agents & how these vary over time Some EC models are of risk aggregation models where estimates for individual categories are combined to generate a single risk figure There may be no best or unique way to do this aggregation Since man

19、y of these assumptions may be untestable, it may be impossible to be certain that a model is conceptually sound While the conceptual underpinnings may appear coherent and plausible, they may in practice be no more than untested hypotheses Opinions may reasonably differ about the strength or weakness

20、 of any particular process in respect of any given property第6页/共36页Validation of EC Models: Introduction to Range of PracticeWhile we will describe the types of validation processes that are in use or could be used, note that the list is not comprehensiveWe do not suggest that all techniques should

21、or could be used by banks We wish to demonstrate that there is a wide range of techniques potentially covered by our broad definition of validationThis is creating a layered approach, the more (fewer) of which that can be provided, the more (less) comfort that validation is able to provide evidence

22、for or against the performance of the model Each validation process provides evidence for (or against) only some of the desirable properties of a model The list presented below moves from the more qualitative to the more quantitative validation processes, and the extent of use is briefly discussed第7

23、页/共36页Validation of EC Models: Range of Practice in Qualitative ApproachesThe philosophy of the use test as incorporated into the Basel II framework: if a bank is actually using its risk measurement systems for internal purposes, then we can place more reliance on itApplying the use test successfull

24、y will entail gaining a careful understanding of which model properties are being used and which are notBanks tend to subject their models to some form of qualitative review process, which could entail: Review of documentation or development workDialogue with model developers or model managersReview

25、 and derivation of any formulae or algorithmsComparison to other firms or with publicly available information Qualitative review is best able to answer questions such as: Does the model work in theory? Does the model incorporate the right risk drivers? Is any theory underpinning it conceptually well

26、-founded? Is the mathematics of the model right?第8页/共36页Range of Practice in Qualitative Approaches to Validation (continued)Extensive systems implementation testing is standard for production-level risk measurement systems prior to implementationE.g., user acceptance testing, checking of model code

27、, etc.These processes could be viewed as part of the overall validation effort, since they would assist in evaluating whether the model is implemented with integrityManagement oversight is the involvement of senior management in the validation processE.g., reviewing output from the model & using

28、 the results in business decisions Senior management knowing how the model is used & outputs are interpreted This should take account of the specific implementation framework adopted and the assumptions underlying the model and its parameterizationData quality checks refer to the processes desig

29、ned to provide assurance of the completeness, accuracy and appropriateness of data used to develop, validate and operate the model E.g., Review of: data collection and storage, data cleaning of errors, extent of proxy data, processes that need to be followed to convert raw data into suitable model i

30、nputs, and verification of transaction data such as exposure levels While not traditionally viewed by the industry as a form of validation, increasingly forming a major part of supervisory thinking第9页/共36页Range of Practice in Qualitative Approaches to Validation (concluded)As all models rest on prem

31、ises of various kinds, varying in the degree to which obvious, we have examination of assumptionsCertain aspects of an EC model are built-in and cannot be altered without fundamentally changing the model To illustrate, these assumptions could be about: Fixed model parameters (PDs, correlations or re

32、covery rates)Distributional assumptions (margins, copulae & shape of tail distributions)Behavior of senior management or of customers Some banks go through a deliberate process of detailing the assumptions underpinning their models, including examination of:Impact on model outputsLimitations tha

33、t the assumptions place on model usage and applicability第10页/共36页Range of Practice in Quantitative Approaches to Validation: InputsA complete validation of an EC model would involve the inputs and parameters, both statistically estimated and notExamples of estimated (assumed) parameters are the main

34、 IRB parameters (PD or LGD) (correlations, PD in a low default portfolio) Techniques could include assessing parameters against: Historical data through replication of estimatorsOutcomes over time through backtestingMarket-implied parameters (e.g., implied vol or correlation, CDS spreads for PD)Mate

35、riality through sensitivity testingThis testing of input parameters could complement examination of assumptions previously & sensitivity testing to described laterHowever, that checking of model inputs is unlikely to be fully satisfactory since, every model is based on underlying assumptionsThe

36、more sophisticated the model, the more susceptible to model error, so checking input parameters will not help here However, model accuracy and appropriateness can be assessed, at least to some degree, using the processes described in this section第11页/共36页Range of Practice in Quantitative Validation:

37、 Model ReplicationModel replication is useful technique that attempts to replicate EC model results obtained by the bank This could use independently developed algorithms or data sources, but in practice replication might leverage a banks existing processes E.g., run a model of the same type or clas

38、s on a the banks data-set However, but once the either the original or test model has been validatedThis technique and the questions that often arise in implementing replication can help identify if: Definitions & algorithms the bank claims to use correctly are understood by staff who develop, m

39、aintain, operate and validate the model The bank is using in practice the modeling framework that it purports to Computer code is correct, efficient and well-documented Data claimed to be used by the bank to obtain its results is in fact being usedHowever, this technique is rarely sufficient to vali

40、date models, and in practice there is little evidence of it being used by banks for either validation or to explore the degree of accuracy of their models Note that replication simply by re-running a set of algorithms to produce an identical set of results would not be sufficient model validation du

41、e diligence第12页/共36页Range of Practice in Quantitative Validation: BenchmarkingBenchmarking the comparison of a banks EC model to alternative models on the banks portfolio E.g., to a vendor model after standardization of parametersAmong the most commonly used forms of quantitative validation used int

42、ernallyA limitation of benchmarking is it only provides relative assessments and provides little assurance that any model accurately reflects reality or about the absolute levels of capital Therefore, as a validation technique, benchmarking is limited to providing comparison of one model against ano

43、ther or one calibration to others, but not testing against realityIt is therefore difficult to assess the degree of comfort provided by such benchmarking methods, as they may only be capable of providing broad comparisons confirming that input parameters or model outputs are broadly comparable第13页/共

44、36页Range of Practice in Quantitative Validation: Benchmarking (continued)There may be good reasons why models produce outliers in benchmarking, all of which complicate interpretation of the results:May be designed to perform well under differing circumstancesMay be more or less conservatively parame

45、terizedMay differ in their economic foundations Comparisons of internal EC are made with varied alternatives:Industry survey resultsRating agency or industry-wide modelsConsultancy marketed models Academic papersRegulatory capital models 第14页/共36页Range of Practice in Quantitative Validation: Hypothe

46、tical PortfoliosHypothetical portfolio testing is an examination of either different banks EC models on a reference portfolio, or different banks EC output from a given reference modelThis is typically a either a reference model or portfolio external to any one bankFrom a supervisory perspective: pe

47、rmits identification of models that produce outliers amongst a set of banks A “model risk management” toolAlternatively, this helps supervisors identify banks that are outliers in risk with respect to a reference model A “bank portfolio risk management” toolIn either case this means comparison acros

48、s banks models against the same reference portfolio (external to the bank) or of banks themselves (their EC for a given reference model) Capable of addressing similar questions to benchmarking, but by different means The technique is a powerful one and can be adapted to analyze many of the preferred

49、 model properties such as rank-ordering and relative risk quantification 第15页/共36页Range of Practice in Quantitative Validation: BacktestingBacktesting addresses the question of how well the model forecasts the distribution of outcomes There are many forms of this that entail some degree of compariso

50、n of outcomes to forecasts, and there is a wide literature on the subjectHowever, weak power of backtesting tests for models of risk that quantify high quantiles has been noted E.g., for portfolio credit models see BCBS (1999)Variations to the basic backtesting approach which can increase the power

51、of the tests have been suggested in the literature: Backtesting more frequently over shorter holding periods (e.g., in market risk using a one-day standard versus the 10-day regulatory capital standardUsing cross-sectional data on a range of reference portfoliosUsing information in forecasts of the

52、full distributionTesting expected values of distributions as opposed to high quantiles第16页/共36页Range of Practice in Quantitative Validation: Backtesting (continued)Backtesting is useful principally for models whose outputs are a quantifiable metric with which to compare an outcomeHowever, some risk

53、measurement systems in use have outputs cannot be interpreted in this way and cannot be backtested Such risk measurement approaches not amenable to outcomes-based validation might nevertheless be valuable tools for banks E.g., rating systems, sensitivity tests and aggregated stress losses. The role

54、of backtesting for such models, if used, would need elaborationIn practice, backtesting is not yet a key component of banks validation practices for economic capital purposes第17页/共36页Range of Practice in Quantitative Validation: Stress TestingStress testing covers both stressing of the model and com

55、parison of model outputs to stress lossesThe outputs of the model might be examined under conditions where model inputs and model assumptions might be stressed This process can reveal model limitations or highlight capital constraints that might only become apparent under stress While stress testing

56、 of regulatory capital models, particularly IRB models, is commonly undertaken by banks, there is more limited evidence of stress testing of economic capital modelsThrough a complementary programme of stress testing, the bank may be able to quantify the likely losses that the firm would confront und

57、er a range of stress events Comparison of stress losses against model-based capital estimates may provide a modest degree of comfort of the absolute level of capital Banks report some use of this stress testing technique to validate the approximate level of model output第18页/共36页Range of Practice in

58、Validation: Additional ConsiderationsWe have not mentioned internal audit, but validation of the overall implementation framework and process should also be subject to independent and periodic reviewThis work should be made by parties within the banking organization that are independent of those acc

59、ountable for the design and implementation of the validation process A possibility is that internal audit would be in charge of undertaking this review process, and as such it could be viewed as comprising a part of the management oversight process discussed previouslyThe list of validation tools al

60、so does not address the issue of adequate internal standards relevant for validation Examples of such standards include:A description of the issues that need to be addressed as part of validation The standards that capital models are expected to achieve A series of quantitative thresholds that models need to meetWarning indicators for particular monitoring metrics Assessment ag