4 ms·

Its funny, and characteristic of that part of the field, that mathematical provability in not even mentioned explicitly. A model (and a learning algorithm) sho

Its funny, and characteristic of that part of the field, that mathematical provability in not even mentioned explicitly. A model (and a learning algorithm) should be interpretable if we can clearly state the assumptions, and prove that under these assumptions, we get what we state we get. We can predict the movement of the planets (short term). This is interpretable, because we only assume a model of 3d space, and Newton's laws(1). The rest is mathematics, which gives us elliptic trajectories, etc. The resulting predictions may be wrong, due to wrong assumptions, but the model is still interpretable. Linear regressions, histograms, decision trees(2) are all interpretable. We know exactly what they do, under proper assumptions. Yes, these models can be sometimes manipulated. But we know this precisely because we know exactly what they do. Whether we should use these models, or statistics in general, in courts etc, is an ethical decision, and perhaps practical decision, similar to whether we should use the death penalty. Has nothing to do with interpretability. With CNNs, at the moment we have very vague understanding of both the assumptions and the models. When we do have this understanding, we will likely also have better and simpler models. Mathematics was always a tool for understanding reality. Proving is understanding. And it was pretty successful so far. (1) and some other things -- the rest of the planets are far enough, etc. (2) given enough data wrt the size of the tree, under proper distributional assumptions.

lasso1 8y ago
Along the same lines, convergence is mentioned as a candidate for interpretability but convexity is not mentioned even once.
- danielmorozoff 8y ago
  This is a very interesting notion, could you elaborate why convexity would be a candidate? Convexity defining a class is clear, but as a proxy for interpretability I have not heard this before. I would assume this would need some measure of degree of convexity - different than strong > strict > convex
mlthoughts2018 8y ago
I don’t agree with this. For example, you can set up a collection of assumptions to underpin frequentist statistics, and then create a set of theorems about consistent or unbiased estimators, and develop a theory like that of p-values. But then in a practical setting, the model doesn’t correspond to something physical or to the inference goal of a practitioner. The p-value tells you something about the relative extremity of a certain statistic that, under certain assumptions, will have a particular distribution. The practitioner wants to know the posterior probability of a particular model or hypothesis given the data, and the frequentist outcome literally can’t comment on it. In this sense I think being able to state assumptions and connect them to outcomes with theorems is good, but not always necessary or even sufficient for applied work. And there can creep in nasty subjective aspects of the problem that are uniquely defined by the specific inference goals at hand. Proofs about how a model would behave under assumptions are often totally useless in these cases. Practitioners don’t use linear regression for complex financial models because the assumptions hold or because of nice properties if the assumptions held. They use them because it’s simple and easy and sort of “just works” despite glaring flaws.
- cosmic_ape 8y ago
  I don't think there's a contradiction. My comment was about what it means to understand the model, or perhaps about what it means to do science, but not about what is necessary or sufficient for applied work. But if we do talk about that though. Yes, science proceeds by modifying or sometimes totally abandoning the assumptions. No silver bullet, and the point is to assume as little as possible. In general, if things "just work", its not a reason to abandon attempts to understand them. Things "just work" until they don't, especially in finance. Consider two guys. Guy A, he predicts that the sun rises every day and is there until dusk. It "works", and so, as the joke goes, he does not touch anything. And guy B, he knows physics and has telescopes. And he can predict the solar eclipses. He knows the failure modes of the model of guy A. Consider the difference between the two.
  - mlthoughts2018 8y ago
    I feel like you’re arguing against yourself here. Guy A’s shallow model would easily be considered more interpretable than Guy B’s sophisticated astronomical physics model, though Guy B’s model is clearly able to articulate more detailed predictions about what might happen. Guy B’s model is less interpretable, but because it “just works” in a vastly greater number of ways (can be applied to the moon, constellations, other planets), people use it or care about it. If all that astronomy offered was a more verbose description of why the sun rises, yielding literally zero different predictions from Guy A’s super simple model, no one would care, and might call Guy B a witch! I’d argue that what matters for science is pure predictive efficacy. That’s it. If you can explain something, it means you can accurately predict something about an unknown state of affairs that would falsify your model if your prediction is wrong. That’s it. Any other kind of explainability is just a matter of linguistic convenience. Of course there is overfitting, etc., but that’s just part of refining the model to yield greater predictive accuracy on unseen data (less generalization error). It’s still all about putting your predictions where your mouth is. If the differing theories of Guy A and Guy B cannot be separated by actually testing the predictions they make, then there is no “interpretability” — just linguistic hand waving. Incidentally I think a lot of modern focus on interpretability is actually about how to be a political gatekeeper or taste-maker through linguistic hand waving, and is not about developing models who better survive the rigors of being required to make real predictions about states of affairs.

lasso1 8y ago

Along the same lines, convergence is mentioned as a candidate for interpretability but convexity is not mentioned even once.

danielmorozoff 8y ago
This is a very interesting notion, could you elaborate why convexity would be a candidate? Convexity defining a class is clear, but as a proxy for interpretability I have not heard this before. I would assume this would need some measure of degree of convexity - different than strong > strict > convex

mlthoughts2018 8y ago

I don’t agree with this. For example, you can set up a collection of assumptions to underpin frequentist statistics, and then create a set of theorems about consistent or unbiased estimators, and develop a theory like that of p-values. But then in a practical setting, the model doesn’t correspond to something physical or to the inference goal of a practitioner. The p-value tells you something about the relative extremity of a certain statistic that, under certain assumptions, will have a particular distribution. The practitioner wants to know the posterior probability of a particular model or hypothesis given the data, and the frequentist outcome literally can’t comment on it. In this sense I think being able to state assumptions and connect them to outcomes with theorems is good, but not always necessary or even sufficient for applied work. And there can creep in nasty subjective aspects of the problem that are uniquely defined by the specific inference goals at hand. Proofs about how a model would behave under assumptions are often totally useless in these cases. Practitioners don’t use linear regression for complex financial models because the assumptions hold or because of nice properties if the assumptions held. They use them because it’s simple and easy and sort of “just works” despite glaring flaws.

cosmic_ape 8y ago
I don't think there's a contradiction. My comment was about what it means to understand the model, or perhaps about what it means to do science, but not about what is necessary or sufficient for applied work. But if we do talk about that though. Yes, science proceeds by modifying or sometimes totally abandoning the assumptions. No silver bullet, and the point is to assume as little as possible. In general, if things "just work", its not a reason to abandon attempts to understand them. Things "just work" until they don't, especially in finance. Consider two guys. Guy A, he predicts that the sun rises every day and is there until dusk. It "works", and so, as the joke goes, he does not touch anything. And guy B, he knows physics and has telescopes. And he can predict the solar eclipses. He knows the failure modes of the model of guy A. Consider the difference between the two.
- mlthoughts2018 8y ago
  I feel like you’re arguing against yourself here. Guy A’s shallow model would easily be considered more interpretable than Guy B’s sophisticated astronomical physics model, though Guy B’s model is clearly able to articulate more detailed predictions about what might happen. Guy B’s model is less interpretable, but because it “just works” in a vastly greater number of ways (can be applied to the moon, constellations, other planets), people use it or care about it. If all that astronomy offered was a more verbose description of why the sun rises, yielding literally zero different predictions from Guy A’s super simple model, no one would care, and might call Guy B a witch! I’d argue that what matters for science is pure predictive efficacy. That’s it. If you can explain something, it means you can accurately predict something about an unknown state of affairs that would falsify your model if your prediction is wrong. That’s it. Any other kind of explainability is just a matter of linguistic convenience. Of course there is overfitting, etc., but that’s just part of refining the model to yield greater predictive accuracy on unseen data (less generalization error). It’s still all about putting your predictions where your mouth is. If the differing theories of Guy A and Guy B cannot be separated by actually testing the predictions they make, then there is no “interpretability” — just linguistic hand waving. Incidentally I think a lot of modern focus on interpretability is actually about how to be a political gatekeeper or taste-maker through linguistic hand waving, and is not about developing models who better survive the rigors of being required to make real predictions about states of affairs.