5 ms·

The Mythos of Model Interpretability in Machine Learning

67 points by alanfranz 8y ago

DanWaterworth 8y ago
You may prefer this link: https://arxiv.org/abs/1606.03490 https://arxiv.org/abs/1606.03490
- black_puppydog 8y ago
  extra points for linking to abstract, not pdf :)
cosmic_ape 8y ago
Its funny, and characteristic of that part of the field, that mathematical provability in not even mentioned explicitly. A model (and a learning algorithm) should be interpretable if we can clearly state the assumptions, and prove that under these assumptions, we get what we state we get. We can predict the movement of the planets (short term). This is interpretable, because we only assume a model of 3d space, and Newton's laws(1). The rest is mathematics, which gives us elliptic trajectories, etc. The resulting predictions may be wrong, due to wrong assumptions, but the model is still interpretable. Linear regressions, histograms, decision trees(2) are all interpretable. We know exactly what they do, under proper assumptions. Yes, these models can be sometimes manipulated. But we know this precisely because we know exactly what they do. Whether we should use these models, or statistics in general, in courts etc, is an ethical decision, and perhaps practical decision, similar to whether we should use the death penalty. Has nothing to do with interpretability. With CNNs, at the moment we have very vague understanding of both the assumptions and the models. When we do have this understanding, we will likely also have better and simpler models. Mathematics was always a tool for understanding reality. Proving is understanding. And it was pretty successful so far. (1) and some other things -- the rest of the planets are far enough, etc. (2) given enough data wrt the size of the tree, under proper distributional assumptions.
- lasso1 8y ago
  Along the same lines, convergence is mentioned as a candidate for interpretability but convexity is not mentioned even once.
  - danielmorozoff 8y ago
    This is a very interesting notion, could you elaborate why convexity would be a candidate? Convexity defining a class is clear, but as a proxy for interpretability I have not heard this before. I would assume this would need some measure of degree of convexity - different than strong > strict > convex
- mlthoughts2018 8y ago
  I don’t agree with this. For example, you can set up a collection of assumptions to underpin frequentist statistics, and then create a set of theorems about consistent or unbiased estimators, and develop a theory like that of p-values. But then in a practical setting, the model doesn’t correspond to something physical or to the inference goal of a practitioner. The p-value tells you something about the relative extremity of a certain statistic that, under certain assumptions, will have a particular distribution. The practitioner wants to know the posterior probability of a particular model or hypothesis given the data, and the frequentist outcome literally can’t comment on it. In this sense I think being able to state assumptions and connect them to outcomes with theorems is good, but not always necessary or even sufficient for applied work. And there can creep in nasty subjective aspects of the problem that are uniquely defined by the specific inference goals at hand. Proofs about how a model would behave under assumptions are often totally useless in these cases. Practitioners don’t use linear regression for complex financial models because the assumptions hold or because of nice properties if the assumptions held. They use them because it’s simple and easy and sort of “just works” despite glaring flaws.
  - cosmic_ape 8y ago
    I don't think there's a contradiction. My comment was about what it means to understand the model, or perhaps about what it means to do science, but not about what is necessary or sufficient for applied work. But if we do talk about that though. Yes, science proceeds by modifying or sometimes totally abandoning the assumptions. No silver bullet, and the point is to assume as little as possible. In general, if things "just work", its not a reason to abandon attempts to understand them. Things "just work" until they don't, especially in finance. Consider two guys. Guy A, he predicts that the sun rises every day and is there until dusk. It "works", and so, as the joke goes, he does not touch anything. And guy B, he knows physics and has telescopes. And he can predict the solar eclipses. He knows the failure modes of the model of guy A. Consider the difference between the two.
jarmitage 8y ago
Anyone have other good references for this topic?
- jclos 8y ago
  Interpretable ML [1] is a small online book on interpretable methods. Even if the content itself is rather shallow, it has links to a lot of more focused papers on the topic. Towards A Rigorous Science of Interpretable Machine Learning is one of the most thorough papers on interpretable ML that I have come across. The main author, Finale Doshi-Velez, has a done a lot of interesting work on interpretability [3]. [1] https://christophm.github.io/interpretable-ml-book/ https://christophm.github.io/interpretable-ml-book/ [2] https://arxiv.org/abs/1702.08608 https://arxiv.org/abs/1702.08608 [3] https://finale.seas.harvard.edu/publications https://finale.seas.harvard.edu/publications
- sibmike 8y ago
  Here is a great caltech lecture about why "simple" models have better out of sample performance. Jump to 28:00, if you want to get straight to the point. https://www.youtube.com/watch?v=EZBUDG12Nr0&list=PLD63A284B7615313A&index=17 https://www.youtube.com/watch?v=EZBUDG12Nr0&list=PLD63A284B7...
Scea91 8y ago
I think interpretability is much more correlated with model size than model type. Small neural net is much more interpretable than decision trees with thousands of nodes.
- taeric 8y ago
  I challenge that last statement. Have there been any neural nets that actually have a solid interpretability? Usually those are more on the lines of effectiveness in training and validation data. With no real clue as to what the driving features were.
  - Scea91 8y ago
    https://distill.pub/2018/building-blocks/ https://distill.pub/2018/building-blocks/ This seems pretty good to me. And the nets are not exactly as small as I meant. I work with random forests and build forest which have more than 80000 nodes per tree. Other than some basic computation of feature importance, it is a black box on the same scale as modern neural nets, maybe even worse.
    - taeric 8y ago
      Thanks for the link. Will take me some time to digest it. Last time I looked at one of these, it was less informative than I'd care to admit. I want to know why a classifier found a vase. The answer is typically some form of "because it was able to see the vase." Would be more interesting to see a classifier that predicts something is going to happen. A predictor that can predict a person is about to step off of a curb, for example. Is it pose of the person? Did it require seeing multiple frames of the person, such that it was an inertia preditor? So, yes, comparing things to gigantic trees can make things tough. But I thought that was the beauty of boosting and building up smaller trees. Most of them are usually more interpretable than you might expect.
      - Scea91 8y ago
        > So, yes, comparing things to gigantic trees can make things tough. But I thought that was the beauty of boosting and building up smaller trees. Most of them are usually more interpretable than you might expect. Regarding your last paragraph, I found this paper https://arxiv.org/pdf/1504.07676.pdf https://arxiv.org/pdf/1504.07676.pdf worth reading. Excerpt from the abstract: "We conclude that boosting should be used like random forests: with large decision trees and without direct regularization or early stopping."
ArtWomb 8y ago
An alternative proposal put out by NYU's AI Now think tank, is to require a kind of "environmental impact statement" for any black box used in government or public sector applications. Instead of rolling out an experimental Predictive Policing agent en masses for example. A "sandbox" is created first. And actual humans in the loop judge if it is prone to bias, unfairness or harm. Public accountability is designed into the algorithm at the outset. https://ainowinstitute.org/reports.html https://ainowinstitute.org/reports.html
relenzo 8y ago
I thought this was a good post! In fact, I had just read a CVPR '18 paper that did the kind of thing he mentioned--presented an 'interpretable neural network' that assumed what interpretability meant...

67 points by alanfranz 8y ago

DanWaterworth 8y ago

You may prefer this link: https://arxiv.org/abs/1606.03490 https://arxiv.org/abs/1606.03490

black_puppydog 8y ago
extra points for linking to abstract, not pdf :)

cosmic_ape 8y ago

Its funny, and characteristic of that part of the field, that mathematical provability in not even mentioned explicitly. A model (and a learning algorithm) should be interpretable if we can clearly state the assumptions, and prove that under these assumptions, we get what we state we get. We can predict the movement of the planets (short term). This is interpretable, because we only assume a model of 3d space, and Newton's laws(1). The rest is mathematics, which gives us elliptic trajectories, etc. The resulting predictions may be wrong, due to wrong assumptions, but the model is still interpretable. Linear regressions, histograms, decision trees(2) are all interpretable. We know exactly what they do, under proper assumptions. Yes, these models can be sometimes manipulated. But we know this precisely because we know exactly what they do. Whether we should use these models, or statistics in general, in courts etc, is an ethical decision, and perhaps practical decision, similar to whether we should use the death penalty. Has nothing to do with interpretability. With CNNs, at the moment we have very vague understanding of both the assumptions and the models. When we do have this understanding, we will likely also have better and simpler models. Mathematics was always a tool for understanding reality. Proving is understanding. And it was pretty successful so far. (1) and some other things -- the rest of the planets are far enough, etc. (2) given enough data wrt the size of the tree, under proper distributional assumptions.

lasso1 8y ago
Along the same lines, convergence is mentioned as a candidate for interpretability but convexity is not mentioned even once.
- danielmorozoff 8y ago
  This is a very interesting notion, could you elaborate why convexity would be a candidate? Convexity defining a class is clear, but as a proxy for interpretability I have not heard this before. I would assume this would need some measure of degree of convexity - different than strong > strict > convex
mlthoughts2018 8y ago
I don’t agree with this. For example, you can set up a collection of assumptions to underpin frequentist statistics, and then create a set of theorems about consistent or unbiased estimators, and develop a theory like that of p-values. But then in a practical setting, the model doesn’t correspond to something physical or to the inference goal of a practitioner. The p-value tells you something about the relative extremity of a certain statistic that, under certain assumptions, will have a particular distribution. The practitioner wants to know the posterior probability of a particular model or hypothesis given the data, and the frequentist outcome literally can’t comment on it. In this sense I think being able to state assumptions and connect them to outcomes with theorems is good, but not always necessary or even sufficient for applied work. And there can creep in nasty subjective aspects of the problem that are uniquely defined by the specific inference goals at hand. Proofs about how a model would behave under assumptions are often totally useless in these cases. Practitioners don’t use linear regression for complex financial models because the assumptions hold or because of nice properties if the assumptions held. They use them because it’s simple and easy and sort of “just works” despite glaring flaws.
- cosmic_ape 8y ago
  I don't think there's a contradiction. My comment was about what it means to understand the model, or perhaps about what it means to do science, but not about what is necessary or sufficient for applied work. But if we do talk about that though. Yes, science proceeds by modifying or sometimes totally abandoning the assumptions. No silver bullet, and the point is to assume as little as possible. In general, if things "just work", its not a reason to abandon attempts to understand them. Things "just work" until they don't, especially in finance. Consider two guys. Guy A, he predicts that the sun rises every day and is there until dusk. It "works", and so, as the joke goes, he does not touch anything. And guy B, he knows physics and has telescopes. And he can predict the solar eclipses. He knows the failure modes of the model of guy A. Consider the difference between the two.

jarmitage 8y ago

Anyone have other good references for this topic?

jclos 8y ago
Interpretable ML [1] is a small online book on interpretable methods. Even if the content itself is rather shallow, it has links to a lot of more focused papers on the topic. Towards A Rigorous Science of Interpretable Machine Learning is one of the most thorough papers on interpretable ML that I have come across. The main author, Finale Doshi-Velez, has a done a lot of interesting work on interpretability [3]. [1] https://christophm.github.io/interpretable-ml-book/ https://christophm.github.io/interpretable-ml-book/ [2] https://arxiv.org/abs/1702.08608 https://arxiv.org/abs/1702.08608 [3] https://finale.seas.harvard.edu/publications https://finale.seas.harvard.edu/publications
sibmike 8y ago
Here is a great caltech lecture about why "simple" models have better out of sample performance. Jump to 28:00, if you want to get straight to the point. https://www.youtube.com/watch?v=EZBUDG12Nr0&list=PLD63A284B7615313A&index=17 https://www.youtube.com/watch?v=EZBUDG12Nr0&list=PLD63A284B7...

Scea91 8y ago

I think interpretability is much more correlated with model size than model type. Small neural net is much more interpretable than decision trees with thousands of nodes.

taeric 8y ago
I challenge that last statement. Have there been any neural nets that actually have a solid interpretability? Usually those are more on the lines of effectiveness in training and validation data. With no real clue as to what the driving features were.
- Scea91 8y ago
  https://distill.pub/2018/building-blocks/ https://distill.pub/2018/building-blocks/ This seems pretty good to me. And the nets are not exactly as small as I meant. I work with random forests and build forest which have more than 80000 nodes per tree. Other than some basic computation of feature importance, it is a black box on the same scale as modern neural nets, maybe even worse.
  - taeric 8y ago
    Thanks for the link. Will take me some time to digest it. Last time I looked at one of these, it was less informative than I'd care to admit. I want to know why a classifier found a vase. The answer is typically some form of "because it was able to see the vase." Would be more interesting to see a classifier that predicts something is going to happen. A predictor that can predict a person is about to step off of a curb, for example. Is it pose of the person? Did it require seeing multiple frames of the person, such that it was an inertia preditor? So, yes, comparing things to gigantic trees can make things tough. But I thought that was the beauty of boosting and building up smaller trees. Most of them are usually more interpretable than you might expect.
    - Scea91 8y ago
      > So, yes, comparing things to gigantic trees can make things tough. But I thought that was the beauty of boosting and building up smaller trees. Most of them are usually more interpretable than you might expect. Regarding your last paragraph, I found this paper https://arxiv.org/pdf/1504.07676.pdf https://arxiv.org/pdf/1504.07676.pdf worth reading. Excerpt from the abstract: "We conclude that boosting should be used like random forests: with large decision trees and without direct regularization or early stopping."

ArtWomb 8y ago

An alternative proposal put out by NYU's AI Now think tank, is to require a kind of "environmental impact statement" for any black box used in government or public sector applications. Instead of rolling out an experimental Predictive Policing agent en masses for example. A "sandbox" is created first. And actual humans in the loop judge if it is prone to bias, unfairness or harm. Public accountability is designed into the algorithm at the outset. https://ainowinstitute.org/reports.html https://ainowinstitute.org/reports.html

relenzo 8y ago

I thought this was a good post! In fact, I had just read a CVPR '18 paper that did the kind of thing he mentioned--presented an 'interpretable neural network' that assumed what interpretability meant...