In a previous ViewPoints article, I discussed the importance of tracking and validating your models. Not only does this help you improve risk management and uncover new business opportunities, it has become a requirement on the part of regulatory institutions.

Fair Isaac recommends that you produce two fundamental types of tracking reports: front-end and back-end reports. This article will focus on the key front-end reports, describing how to prepare them and interpret results. Front-end reports provide the first insights into how well your models represent the observed applicant population and whether there are opportunities for automating more of your decision process. In a future issue of ViewPoints, I will discuss how back-end reports track your model’s predictions against actual performance to help you fine tune strategies and identify model degradation before it becomes a major issue.

Essential front-end reports

The main purpose of front-end reports is to uncover any major discrepancy between the current applicant population and the development population. These reports can also help you identify potential problems with the quality and consistency of the input data.

Front-end reports should be produced quarterly to stay on top of population shifts and aggregated over a year time frame to minimize seasonality effects. If there is sufficient data to make meaningful statistical conclusions, these reports can also be generated at the subpopulation level for greater detail.

Population Stability Report

The Population Stability Report answers the question “Is my population scoring differently than the development population?” These changes could have important consequences for your acceptance rate and, thus, impact the portfolio’s bad rate.

Within this report, the population stability index (PSI) measures the changes in the score distribution of current applicants compared to the development sample, or another baseline measure. By detecting any population shift, it can serve as an early warning of the model effectiveness on that population.

To produce this report, you need the application date and final distribution report from the development sample or your baseline period.

The Figure 1 graph compares results from two consecutive Population Stability Reports. The PSI for each report indicates the level of change in the applicant population. This index can usually be interpreted as follows:

• 0.100 or less: satisfactory
• 0.100 to 0.250: warning; watch for continuing increases in subsequent index calculation
• 0.250 or more: indicates a shift that may be the result of major changes in your population

Figure 1 compares results from two Population Stability Reports, which measure changes in the score distribution of current applicants. Applicants in 2003 are scoring lower. A population stability index (PSI) of 0.3354 indicates a major shift in applicant scores.

The graph show that the PSI can change with shifts in the distribution of the applicant population by score ranges. The applicants received in the first quarter of 2002 had a score distribution similar to those in the model development. The PSI of 0.0016 reflects this.

But in 2003, the situation changed. A higher percentage of applicants are clearly scoring lower. The PSI is 0.3354, indicating a major shift in applicant scores.

Remember that changes in the applicant score distribution may affect your acceptance rates. You may want to raise or lower the cutoff score to adjust for the change.

Using this report alone, it is not clear what impact a shift may have on the quality of your accepted accounts. You should prepare a Characteristic Analysis Report to detect the reasons for a shift.

Characteristic Analysis Report

It’s usually not enough to know that your current population is scoring differently. You also need to know why.

The Characteristic Analysis Report answers “How has my applicant population changed since development?” It can also help you uncover the reasons behind a shift—for example, changes due to a different demographic mix in a region or new marketing campaign.

The Characteristic Analysis Report helps identify possible causes for any shift in the applicant score distribution. It determines which characteristics are scoring differently at the attribute level, and how many points are being added or lost compared to the baseline for each characteristic.

To produce this report, you need the application date, the percentage of applicants scoring within each attribute, the score points associated with each attribute and the percentage of applicants scoring in each attribute for a baseline. As with the Population Stability Report, the baseline may not correspond to the development statistics.

One report should be produced for each characteristic in the scorecard. The sum of the score differences for each of the characteristic’s attributes is the average number of points gained or lost compared to the baseline.

Generally, a large total score difference indicates a significant change in the applicant population that should be explored further. Since positive and negative score differences tend to cancel each other out at the characteristic level, large differences at the attribute level should also be investigated.

Final Score Report

When a change in your acceptance rate cannot be explained by shifts in applicant scoring, score overrides may be to blame. The Final Score Report helps you determine how frequently overrides occur and how they have affected your overall acceptance rate. This information may also identify whether there are opportunities to automate more decisions within specific score ranges.

The report specifically measures the current operational adherence to the scorecard, the current approval rate and the potential approval rates that would have resulted at a particular cutoff in the absence of overrides. This report helps you analyze the effects of factors outside the scorecard, including those related to your overall credit policy.

To produce this report, you need the application date, the final applicant score and the decision made regarding the application (accepted or rejected).

Figure 2 shows a sample Final Score Report. Note the potential and actual acceptance rates. Since the cutoff score is fixed at 200, if there were full operational adherence to the scorecard, the acceptance rate would have been 44%. However, since there were some score overrides, the acceptance rate dropped to 43%.

A Final Score Report analyzes the effects of factors outside the scorecard, such as your override policies. Here, using a cutoff score fixed at 200, full adherence to the scorecard would have resulted in a 44% acceptance rate. Due to score overrides, the actual acceptance rate is 43%.

This report also helps you determine override rates. Low-side overrides refer to the applications that were accepted even though they had scores lower than the cutoff. High-side overrides are rejected applications with scores higher than the cutoff.

As a general rule, the number of high-side overrides is usually higher than the low-side overrides. If the percentage of low-side overrides is significant or unexpected in relation to you credit policies, you should re-evaluate your override policy and verify that the policy is being applied correctly.

Keep in mind that applications that are rejected or accepted based on company policy should not be considered overrides. Instead, they should be tracked separately.

Override Report

Knowing that your business is not adhering to scorecard recommendations is critical.  You also need to know the reasons for the overrides and if there are trends in your override decisions so policies and strategies can be updated accordingly.

To produce this report, you need the application date, override reason and the decision made regarding the application (accepted or rejected). The simple format of this report makes it easy to interpret.

The reasons used to decline applications should be examined carefully to make sure that you are not turning away potentially good customers. US lenders should make certain to have sound adverse action reasons in order to respond to regulatory inquiries.

Reasons for low-side overrides should also be analyzed since these applicants are becoming part of your portfolio. You’ll want to make sure that the override criteria are reasonable and approved by someone at an appropriate level.

Finally, it’s a good idea to track all information used in your override decisions so that you can include or exclude the right data in future model developments.

By regularly producing and reviewing front-end reports, you can react faster to emerging population changes. This key information lets you modify approval strategies proactively instead of waiting for unpleasant changes in approval and/or delinquency rates to show up in future financial reports.

If you have questions, or would like to review your current tracking and validation procedures, we can help. Many clients have benefited from extra support in this area through training, periodic review of reports, independent model validations and other Fair Isaac services.

Santiago Libreros (slibreros@fairisaac.com) works on Fair Isaac’s Predictive Science team.