Data Science & AI

What You Don't Get With a World Class Modeling Tool

Zest AI team

May 7, 2018

The dirty little secret of machine learning is that the algorithms are the easy part. Not to say that math is simple, but rather that there are great books1 and tools2 to help you build just about any model you want. Instead of worrying about the algorithm, per se, you should worry about how to get that model into production.

In some cases, getting a model into production requires rewriting it from your modeling tool into a language, like JAVA or Ruby, that you run into production. That’s long and hard, often taking months to get correct and complete. Increasingly, however, modeling tools3 provide the ability automatically to promote your models from your modeling environment into production.

Although important, I view this as table stakes to the real challenge: Do you really know what your model does? Model validation is critical for all modelers, so you understand when and under what circumstances your model will work as predicted, and when not. This knowledge allows you to know when you want to trust your algorithm, and when you should do something else. In financial services, you have a higher bar: You have to be able to prove why you made each decision to regulators and customers.

Most machine learning (ML) algorithms are black boxes; they cannot provide that proof. There is a great deal of focus, at the moment, on methods to turn those black boxes into open ones. There are several approaches that work, in various degrees. Without one, financial services adoption of ML will slow, or stop.

Some approaches seem plausible, but actually don’t pass regulatory muster: Models that proxy an explanation with a simpler model fall into this category. The two problems with so-called “proxy models” are that first, the model gives up much of the power of the ML parent model, and second, that features derived from an ML model are not explainable. Thus, you give up some power and lose key explainability. There are other approaches that pass muster, but may seem implausible or computationally impossible. For example, some explanation tools use permutations of the actual model (e.g., “leave one out” methods) to generate explanation. Sounds great, except if you have 100 signals in your model, you may have to recompute the model 100 factorial times for each decision. 100 factorial is equal to a 10 followed by 150 zeroes. Could take a while.4

But regardless of the modeling technique or validation method you choose, the thing you don’t get from most awesome modeling software is the ability to get your model into production, with all the explainability tools, and the help to complete all the documents and artifacts you need to hand to your regulators, model governance team, and management. Only ZAML gives you that.

Douglas Merrill is the co-founder and CEO of ZestFinance. He spent 5 years at Google as their CIO and VP of Engineering. Douglas is a technologist and data junkie at heart, but also loves motorcycles, tattoos, and ’80s music. He cares deeply about people and animals and supports a variety of charities related to drug rehabilitation and animal rescue.

Thank you for subscribing!
Something went wrong while submitting the form.