The adage “choice is the enemy of decision” certainly holds true for optimization solutions. As banks move toward optimizing customer decisions, the many choices available make it difficult to know what works best.
While some optimization solutions are purely software-based “engines,” others offer more, such as modeling services or domain expertise within a given decision area. And choice is not your only challenge. Historically, many solutions are highly academic, and not built to meet real-world requirements.
So, what are the right criteria to evaluate an optimization solution?
Start by asking the following 10 questions, designed to help you understand the requirements—and avoid the pitfalls—of decision optimization. These reflect lessons learned from Fair Isaac’s extensive work with firms successfully using optimization today.
1. How does the solution incorporate the sensitivities of consumers to various actions?
In order to optimize actions that you take on your customers and prospects, you first need to understand the reactions of each consumer to possible treatments you can take. Optimizing based on incorrect assumptions about these “action-effect” dynamics will render the optimization ineffective (at best) or grossly incorrect (at worst).
Incorrect models of consumer behavior—whether on the dimension of response to a product offer, risk from a line increase or revenue from a pricing action—lead to incorrect optimization results. In other words, “garbage in equals garbage out.”
Consider Figure 1, which shows three curves representing the relationship between expected profit and different levels of loan price for a particular consumer.
The orange curve represents the “true” relationship between profit and different levels of loan price. Note that a consumer’s price sensitivity is typically not known a priori for several reasons—for instance, you may not have tried every possible combination of price on this consumer, actions other than price are taking place simultaneously, or the environment is changing. This true relationship only becomes apparent after the actions are taken, and balances the likelihood of taking up the loan with the potential revenue and loss.
The red line represents a “good” estimate, since it estimates the true relationship closely. The green line shows an incorrect estimate that is a bit off, perhaps due to data biases.
Without constraints, the optimal action is determined by the top point on the curve. If we took action based on the incorrect estimate, we would offer this consumer 9.49%, while an action based on the good estimate would offer 10.49%. Looking at these points on the true profit curve, the good estimate would increase profit by about $4 per account. Multiplied over hundreds of thousands or millions of accounts, the effect of the incorrect estimate on profit would be significant.
Action-effect models are typically more complex than the examples above, since they are influenced by multiple factors. The modeling techniques required to accurately understand consumers’ sensitivities are very different from traditional scorecard approaches. Ensure that your vendor is fluent in decision modeling techniques, can validate its assumptions and approach, and can deal with limitations in data.
2. Does the optimization solution address data limitations?
Getting the action-effect relationship right depends on both data and business expertise. Data will never be perfect, and any vendor who claims you must invest 6 to12 months in running experimental designs to gather perfect data is mistaken. Leveraging your existing data is important, and can tell you a lot about your customers and their response dynamics.
The ability to identify holes and/or biases in data is an important component of your solution, since these can lead to incorrect assumptions about consumer behavior, as illustrated in question 1.
In the rare cases where no data is available, there are still ways to move forward today, including: 1) developing expert models by encoding business expertise into the assumptions and relationships, and 2) performing smart, limited testing to quickly gather the data to inform your models. In cases where you need to rely heavily on expertise, more stress-testing may be required, which we’ll discuss further in question 7. Choose an optimization solution that explicitly identifies and addresses data limitations, and can incorporate business expertise.
3. Can you leverage your existing analytic assets within the solution?
You have invested in gathering data, building predictive models and perhaps even building action-effect models to drive your optimization solution. The optimization framework you choose should allow you to incorporate these models in a way that you’re confident all relevant customer data is utilized to make the best decision. Your data elements and models can be used as inputs or decision keys, and in many cases, can be used directly in the underlying action-effect models themselves.
4. Does the solution provide insight into your key business trade-offs to facilitate the selection of an optimal operating point?
Your business will need to optimize decisions subject to many constraints. These range from policy constraints, such as whom you lend to or what offers certain customers are eligible for, to portfolio-level goals, such as “reduce losses by 5%” or “ensure marketing budget is less than $10MM.”
While some optimization solutions focus on finding the “one” solution to a problem you specify, it is always important to understand the impact business constraints have on your bottom line and your strategy. Exploring these trade-offs helps to properly set those constraints.
A key tool for quantifying these trade-offs is an efficient frontier, as seen in Figure 2. For example, if you want to both increase profitability and market share, you may look at the rate at which new loans are “funded” (approved and opened).
If today you are operating at point A, you can both increase profit and market share by moving to the efficient frontier (e.g., move up and to the right to point C).
If you are operating at point B, the trade-off between business goals really comes into play. You may decide to maintain your current market share and increase profit (by moving up to point D). Or you may decide to maintain or even sacrifice profit to gain additional market share (by moving to points E or F).
Similar trade-offs can be made between profit and loss, volume, attrition, exposure or any other business metrics of interest.
A trusted advisor with relevant business expertise can help you identify the optimal operating point. Make sure your optimization vendor can provide someone you can work with to identify options and make a choice on strategies to deploy.
5. Does the solution allow you to establish and evaluate a baseline of “business-as-usual” or what you’re doing today?
There are many reasons why you should be able to measure your current strategies in the context of the optimization solution.
First and foremost, it provides an ROI estimate of the optimized solution. It’s important for this estimate of lift to be made within the optimization solution to ensure an “apples-to-apples” comparison. It also provides a sanity check that your decision model is able to project the results of the historical strategy.
Another reason is to tie the various outcomes and constraints to what you’re experiencing today. For instance, “reduce attrition by 10%” will only be meaningful if you tie it to the current outcome.
In some cases, the business-as-usual baseline is best represented directly from the data. These historical actions can be passed directly through the simulation. This option tends to work best when multiple strategies (e.g., champion/challenger) are being used.
In other cases, it is helpful to represent your baseline strategy as a set of rules or a decision tree, particularly when this strategy is not represented in the data.
A good optimization solution will provide both options for simulating your current strategy or even a strategy you are considering.
6. Does the solution provide a mechanism for business users to verify that the optimized solution is valid?
Optimization results are only as good as the assumptions and inputs on which they were built. As such, it is imperative to evaluate the optimization results from both a technical and business point of view.
A mechanism to investigate optimization results allows the business user to impute business judgment into a process that is otherwise a pure mathematical exercise.
Let’s look at an example for offering a new line of credit. Figure 3 compares customer profiles generated by an optimization solution: customers who are rejected, accepted with a low line and accepted with a high line. The goal is to see whether the more favorable treatments will generate the profit improvements that the solution projects.
Business experience tells us that example 1 may be too optimistic. For instance, the “accept with high line of credit” group is still fairly risky (judging by FICO® score and debt-to-income ratio) and their need for credit is relatively low (37% utilization). It seems unlikely that our solution would generate many gains from this treatment.
In example 2, the solution identifies less-risky accounts with higher credit needs for the more favorable treatment. Therefore, it is more likely to generate profit.
The ultimate question is: Can the business user justify the treatments based on his/her business knowledge? In other words, is the optimization driven by unreasonable modeling assumptions?
To help the business user validate the reasonableness of the approach, the optimization solution should have reporting functionality that provides visibility and hands-on review. Swap-set reports can also be useful to compare different optimized strategies, or an optimized strategy, to what you’re doing today.
Ideally, the business user could create his/her own diagnostic reports in the tool and drill down where necessary to get comfortable with the optimization.
7. Can you stress-test changes in the environment within the optimization solution?
Despite investing in good data, action-effect models, software and expertise, unexpected events can happen to change the world in which we’re doing business. The economy changes, the competitive environment changes, the way consumers react changes. This means the outcomes we were expecting from our strategy—such as profit, revenue, loss, market share—will also change.
The key to anticipating these changes, and choosing strategies that will fare well under adverse conditions, is not only important from a regulatory perspective (as required by the Basel II Accord), but it makes good business sense.
An optimization solution must provide a facility to perform what-if scenario analysis that goes beyond simply changing your constraints. It should be able to simulate the impact of changes, including external factors (such as the economy) or consumer behavior (such as price sensitivities or risk levels).
The best solutions will allow you to: 1) automate stress-testing to perform many scenario investigations at once; 2) assume that changes may apply differently to different segments of your population; and 3) provide a range of insightful diagnostics and support to inform your choice of optimal strategy, in light of what the future may hold.
8. Can the optimization framework be easily adopted for new decision areas across the enterprise?
Some optimization frameworks require the user to formulate the mathematical expressions corresponding to the optimization problem. While applications made for a mathematically oriented “super user” allow flexibility, the trade-offs are that it can be more difficult to use, further from the business problem and often very error-prone. Using this methodology, it is much harder to validate mathematical expressions than to validate simple business logic. In addition, it is much more difficult for the business user to understand and use the framework, and may require specialized training or additional resources.
Conversely, if the framework is too restrictive, you will not be able to leverage it in other applications and decision areas, which is a common goal for many building an optimization practice. For example, you might want to optimize your marketing decisions after you tackle your line increase decisions.
Therefore, it is critical to balance flexibility with skill set requirements when choosing an optimization solution, and have a tool that both meets analytic requirements and communicates to the business user. This will allow your organization to scale your optimization efforts from a common methodology and platform, and develop a repeatable process that can grow as your business grows.
9. Can the optimization solution be readily deployed in your existing decisioning platform?
After choosing the optimal scenario to be deployed for your portfolio, the next step is to bring those optimal actions or rules to your current decision execution platform. In financial services and elsewhere, this is most often in the form of a strategy tree that can assign treatments to current and future customers. However, in certain circumstances, it may mean calling the optimization from within an application to calculate optimal decisions in real time.
If you are deploying to a rules management system, it’s beneficial to have software that can export these rules in a format that your system can accept, such as code. This step eliminates the errors that often happen when trying to manually enter the optimized strategy, which can be much more complex than a typical strategy.
The key to deployment is flexibility and the ability to translate optimal actions to be used by your system.
10. Do you have sophisticated optimization software designed to solve financial services business problems?
Optimizing decisions on millions of customers with multiple business constraints—as we do in financial services—requires an optimization algorithm that can find optimal solutions efficiently. Your optimization algorithm must solve for the best (i.e., true optimal) solution and exploit the structure of your decision problem to do so efficiently.
Questions to ask your vendor include:
Many optimization software vendors try to circumvent the problem by optimizing at the segment level or using an approximate search algorithm—relaxation of the optimization problem, as it is known in academia. The best optimization algorithms can exploit the structure of a particular decision problem, while still solving for the true, rather than approximate, solution.
In financial services, it is reasonable to assume a problem size of millions of consumers or accounts, with dozens of business constraints (in practice, we see two to five portfolio constraints most typically applied), and dozens to hundreds of possible treatments. Depending on your computing power and number of constraints, this size problem should take a few minutes to a few hours to solve.
Finally, choose a vendor that continually invests in its optimization algorithm to improve performance and efficiency.
High-value optimization
Evaluating an optimization solution using the guidelines above can help you avoid potential pitfalls and ensure your solution delivers true business value. These 10 questions were crafted using best practices from Fair Isaac’s nearly 100 optimization projects across different countries, portfolios and applications.
For information on Fair Isaac optimization solutions and case studies, contact us at 1-888-FIC-6336 or analytics@fairisaac.com, or visit www.fairisaac.com/StrategyScience.