Model validation is the set of processes and activities intended to verify that models are performing as expected, in line with their design objectives and business uses [US Federal Reserve]. It is a critical part of model risk management.
Model validation should be performed prior to deployment and routinely after deployment as part of the monitoring process. It should also be performed when usage context changes, changes are made to model implementation or too much time has passed since the previous model validation.
Goal #
- To ensure that models are sound by identifying potential limitations and assumptions and assessing their possible impact.
Tasks #
- Evaluate the extent to which the model satisfies the business objectives
- Run the model in a real business application in a test environment and compare model outputs to corresponding actual outcomes
- Examine model results that are not necessarily related to the business objective to reveal additional challenges, information, or hints for future directions
- Review the model building process and identify activities that may have been overlooked, those that should be removed and those that should be done again
- Evaluate the model on synthetic scenarios to determine the conditions the model can be used (stress-tests)
- Determine next steps depending on the results of the assessments, i.e, deploy the model, repeat certain steps, abandon the project, or set up a new project
- After deployment, benchmark model’s inputs and outputs to estimates from alternatives to verify that the model is still valid
Roles #
- Model validation should be carried out by a group independent of the team that built the model.
