Credit & Risk
Why Custom Credit Scores Are Better Than Generic Ones
June 25, 2021
A credit union executive texted me the other day to complain about his own institution’s underwriting practices. A member of his credit union had just been declined for a $50,000 home improvement loan. Even though the applicant’s debt-to-income ratio met the lender’s threshold, the debt-to-payment number was a hair too high. Declined.
“Isn’t that a little harsh?” he wrote. 100%, I texted back. Home improvement loans rarely default, and the member had a prime credit score.
And that’s what was so frustrating for the lender. “There is no confidence in credit scores right now,” he wrote back. While the member finally did get approved after hours on the phone, no one ended up happy. “We’re walking away from good deals all the time.”
“There is no confidence in national credit scores right now. We’re walking away from good deals all the time.”
Lenders are finding it hard right now to assess consumer credit risk using the generic, national scores due to massive employment volatility and federal stimulus. In December the Federal Reserve’s Advisory Council, a group made up of the nation’s biggest banks, reported that “many Council members expressed concern regarding the difficulty of assessing customer risk using traditional methods.”
“For example,” the Fed wrote, “FICO scores do not provide meaningful insights as the average score in the third quarter  was about 711, which is the highest since FICO started tracking in 2005. Similarly, credit insight into small businesses, which is traditionally score-based, has also been challenged.” Stanford researchers are also warning that generic credit scores are too imprecise to help millions of underserved Black and Hispanic borrowers get affordable loans.
This is a bad time to be losing confidence in legacy underwriting techniques. A flood of deposits has placed tremendous pressure on lenders to expand their loan portfolios. For many, that means growing by moving down the credit spectrum. Some lenders are even wondering how to serve the 50 million thin-file and no-file consumers considered unscorable by the generic models.
For those lenders looking to grow, now is the time to consider switching from generic credit scores to custom machine learning models. Custom ML models use better math and 10x to 100x more variables from standard credit bureau files than are used in a national generic score. They can also be built using performance data from both funded and unfunded populations in a lender’s geographic market to ensure the lender is not creating blind spots with their model.
ML models, with their higher statistical precision than generic scores, are designed to find more good loans in a sea of uncertainty. While the national scores may be good at picking out prime borrowers, that’s not exactly hard to do. Where generic scores struggle is at predicting creditworthiness once you go below prime. The chart below compares the risk-ranking ability of a Zest-built ML model (green line) to a national score (black) for a major subprime auto lender. The national score goes sideways (indicating an inability to accurately assign risk) in the middle quantiles of the default-risk curve. The Zest model is more certain where applicants belong all the way across the risk spectrum.
With greater accuracy comes greater confidence to automate credit decisioning. Better risk-ranking shrinks this noisy middle. A bigger portion of scores on the low or high end of the spectrum can be automatically denied or approved, shrinking the number of apps in the middle that need to be sent to manual review, greatly reducing operational costs.
This more expansive use of bureau data (that lenders are already paying for) gives lenders the ability to swap in the low-risk borrowers with lower traditional scores. Not all 620s are the same. Some are on their way to 700 and some are on their way down from 700. Only a custom ML model can capture the nuances and trends that reflect these movements. The chart below illustrates the impact an ML model has on moving borrowers with low credit scores up into the approvable segments -- it sees they will perform as well as the high scoring borrowers.
Stability Over Time
Models are only as good as the predictions they make, and custom ML models can be tough for even the best data scientists to make stable when confronted with changing economic conditions. Zest’s model development tools have been designed to address this issue. From data cleaning to strict separation of development from test data, Zest tools produce models that are stable despite changing economic conditions. A study by Experian that looked at the performance of FICO scores during difficult economic times showed that actual default rates could differ from the expected default rate by 50-150%. That’s quite a lot of variance to absorb as a business.
A custom risk score holds up better in the face of changing economic times. Zest customers have had their models live throughout the COVID cycle and they continue to rank order risk accurately. One customer built challenger models with more recent data in them, using Zest software, to test this hypothesis. When compared with the new models that incorporated more recent observations, the old models Zest launched years ago performed just as well. The ability for custom models to hold up in uncertain times has been proven here in the U.S. and in international markets among our customers who have had models deployed in some of the most tumultuous economies in the world.
Besides their precision challenges right now, generic scores are also struggling to deliver on lenders’ inclusivity goals. Credit data, in general, suffers from the impact of generations of racial and gender discrimination. A generic linear model based on a couple of dozen variables can do little to remove embedded bias, and can often generate an unhealthy amount of disparate impact across races. An Urban Institute report using 2016 data from Freddie Mac shows that FICO scores are correlated with race. Unfortunately, lenders can lawfully justify using a biased model if they can prove there are no less discriminatory alternatives.
In search of fairer solutions, lenders are turning to alternative data sources (such as internal cash flow data) or custom ML models that use advanced math to dial down the influence of variables that cause disparate impact. The days of using business justification to default back to a biased generic score are coming to an end.
At Zest, we’ve developed a workflow that produces more precise ML underwriting models that also shrink the approval rate gaps between protected and unprotected classes (also known as the adverse impact ratio). Here’s an illustration of what that looks like in action. The lender’s original logistic regression model (which are often simply a national generic score) is significantly improved in accuracy (by 200 basis points) by moving to an ML model. Zest’s automated de-biasing software kicks into action, optimizing the most accurate ML model for fairness and reducing the adverse impact ratio (left axis). Lenders can choose from a range of models that are more fair and more accurate. The chart below shows slight sacrifices in incremental performance to achieve more inclusion. Some lenders may find less discriminatory alternatives with no sacrifice in performance. None of this is possible with a generic score.
It may seem like national credit scores have been around forever, but they’ve only been in use since the early 1990s. Prior to that, many broader methods were used to determine if someone was trustworthy, many of which were closer to what ML scores like Zest’s do today – taking in many different forms of information, often taking into account local considerations.
Custom models are now cheaper, easier to create, and more powerful than ever thanks to the advent of machine learning-based underwriting. For most lenders, custom ML underwriting models tuned to a specific customer base, geography and product, are becoming a better option than generic scores. They deliver better performance, maximize investment in data, and take advantage of your local market knowledge.
November 24, 2021
Zest AI team
November 18, 2021
November 16, 2021