Five questions to ask your AI vendor about explainability
A wave of new fintech companies are pushing traditional lenders to consider new technologies in order to stay competitive
One of the most promising technologies on the market today is called advanced machine learning (ML), which is — at its core — a branch of AI that uses mountains of data and sophisticated math to help lenders make better decisions and predictions.
Before making any investments in ML, lenders need to ask the right questions in order to capture the gains.
Take the issue of explainability, for example
By law, lenders must be able to show why they approved or denied a loan.
In the old days (okay, sure some lenders are still using these methods) that was easy. Typical underwriting models had only 30 or so variables, and none of them interacted with each other, making it clear which ones were driving the decision. But ML models analyze thousands of factors as well as the interactions between them — meaning it's pretty impossible for humans to easily determine which variables are driving the decision.
This problem, often referred to as a 'black-box,' has attracted a handful of techniques that say they can extract the top reasons for loan approvals or denials, such as LIME, permutation impact, and leave-one-column-out. But in recent tests, we found these methods lacking in ways that could expose you to risk.
None of them performed as well as the explainability baked into Zest AI's technology, which uses a different explanation method, derived from competitive game theory and multivariate calculus, that generates explanations from the actual underlying model. To capture the benefits of ML, it’s crucial that lenders switch to ML underwriting that gets consistent, accurate, fast explanations from credit model decisions.
Here are five key questions to ask any AI technology vendor before you make any decisions about deploying ML in your business. If your vendor can’t provide a clear, unequivocal “yes” to each of these questions and explain how their technology meets these criteria, it may be time to look for a new AI vendor.
1. Can your explainability technique run and chew gum at the same time?
Most explainability techniques can analyze the effect on a model of only a single variable at a time. With hundreds of variables in an ML model, that kind of brute force work can drag on for hours or days. And without capturing the interactions between and among multiple variables, something most explainability techniques are incapable of doing, your explainer will compromise on accuracy.
2. Does your technique get under the hood?
Most explainers hazard a guess as to what’s under the hood of an ML model by looking at what data goes in, and what comes out. They don’t examine the model’s internal structure to understand the interactions between the many variables. That means you’re getting much less information than with explainers that delve into the model. You might be gambling with accuracy.
An example that shows why this matters: An explainer that looks at only a section of a model’s data space might fail to see bias or discrimination and conclude that no disparate impact exists, which may be true. But because you’re only looking at a small section of the model, there is no guarantee that disparate impact doesn’t exist somewhere else in the complexity of the real model. Missing such an important detail due to a lack of accuracy in your fair lending analysis raises serious ethical and regulatory risk once the complex model is deployed to make real-world decisions.
3. Is your explainer playing make-believe?
You need to know if your explainer is looking at the final model because that’s the only one that matters. You might get some insights from analyzing works in progress, but again, you’ll be compromising on accuracy. Some techniques also probe the ML model with non-real data. But you don’t want an explainer that uses synthetic or randomly shuffled data, because that leads to inconsistencies.
4. Is this explainability technique a consistent performer?
Consistency is key for underwriters. Every time a model scores a specific borrower application, it should yield the same prediction, and if there’s a denial, it should yield the same set of primary reasons for the denial. Every explainer used for lending should return the same answer when the input is the same. If you don’t have that, you might get different explanations from analyzing the same borrower twice. For example, if you re-build your model using the same data, your explanations should remain the same. This seemingly trivial property is not true for most explainability techniques.
5. Does your explainer see the bigger picture?
Underwriters need their ML explainability technique to more holistically understand a model, which explainers looking at local “slices” of a model often fail to do. Let’s use a basketball metaphor: if you were trying to explain to a Martian why Kobe Bryant is a great basketball player, you’d probably mention that he’s taller than most humans. But an explainer such as LIME, which gets its answer by looking at a small piece of the model, might see a lot of other NBA players, who are also going to be tall, clustered around Kobe and thus determine that his height is not so much of a differentiating factor — when indeed it is, compared to typical humans.
