4 ms·

I don't think what you are saying is incompatible with my comment. Asking a question such as, "what is it about the data that is causing the model to be fitted

by p4wnc6 10y ago

I don't think what you are saying is incompatible with my comment. Asking a question such as, "what is it about the data that is causing the model to be fitted like this?" is very healthy and important, but is rarely what anyone means by 'interpretability.' Instead, 'interpretability' is usually taken to mean something more akin to: which readily human-understandable cognitive structures (out of the set of such structures that I am the authortity on) is the model directly (and easily) mappable to? By mapping onto those structures, I can now weave a story about the model within the domain in which I am the established authority. If such mappings don't exist, or are very difficult to compute, then my authority is threatened. Having said all of this, the most egregious abuses of the nefarious side of 'interpretability' that I have seen have all come in quant finance, which is a situation in which the optimization criteria are literally the identically same thing as the actual objective. Given that when prediction accuracy is paramount to the definition of success you still see politics trumping science, I have very little confidence that when you move to domains where the objective criteria of the models differs from the real-life objective in non-trivial ways, then somehow 'interpretability' will be used appropriately rather than overwhelmingly just as a political tool.

radicalzebra 10y ago
With respect, I think there's a fundamental error in this reply. You're making a dogmatic statement about what interpretability is taken to mean, which assumes of course that the vast body of work, both in academia, journalism, and business meetings, holds a consistent view. The point of this paper, however, in part, is to point out that this view is mistaken.
- p4wnc6 10y ago
  You are incorrect in your claim. The paper does make a good point that under the umbrella term "interpretability" there are varied and conflicting understandings of what, specifically, is meant. That fact has nearly zero relevance to what I'm saying. When I say that people look for mappings from the model to a space of cognitive structures over which they have authority, it could be (and observed in practice always is) a smattering of all the different things talked about in the paper -- authority over which diagnostics represent 'trust,' authority over which visualizations are adequately informative, authority over explaining 'how' an algorithm works, and so on. Though all of these are discordant takes on the specifics of interpretability, as discussed in the paper, they still are unified in the sense that the true intention in appealing to any of them is to map things into a domain where someone wields the authority. It's not about actually caring about trusted diagnostics, or caring about quality visualizations, or caring about communicating the intuition behind a model. Those are just the plausible deniability excuses proffered to try to win the battle over which faction's authority gets center stage. I don't require that anyone have a cross-domain consistent view of the specific definition of interpretability in order for my claims to be a viable explanation, and so it is perfectly compatible with the paper. The paper looks at the on-the-ground specifics used in the arguments over interpretability, meanwhile politics explains the behind-the-scenes motivations of any of those specific choices. The Moral Mazes chapter Dexterity with Symbols is a good place if you want to read on the evidence about how this is carried out in management bureaucracy. It has actually been formally studied in some interesting ways.
moultano 10y ago
> Asking a question such as, "what is it about the data that is causing the model to be fitted like this?" is very healthy and important, but is rarely what anyone means by 'interpretability.' What I see people mean by interpretability is one step back. It's literally just "what is the model actually doing?" in few enough concepts that meat machines can hold them in memory. Sometimes accompanied by a second question "is the thing it is doing dumb or smart?" In my field (search) there are large classes of things that "work" but are dumb or annoying, or just not the product we want to provide, and we have to make product decisions to avoid them. So in that context figuring these things out is really important. >Having said all of this, the most egregious abuses of the nefarious side of 'interpretability' that I have seen have all come in quant finance, which is a situation in which the optimization criteria are literally the identically same thing as the actual objective. You know more about this than me, but it seems like there might be big differences between models in how they respond to unquantifiable downside risk of rare events. How do you deal with that without interpretability? How do you examine the assumptions the model is making that you may not want it to make?

by p4wnc6 10y ago

radicalzebra 10y ago

With respect, I think there's a fundamental error in this reply. You're making a dogmatic statement about what interpretability is taken to mean, which assumes of course that the vast body of work, both in academia, journalism, and business meetings, holds a consistent view. The point of this paper, however, in part, is to point out that this view is mistaken.

p4wnc6 10y ago
You are incorrect in your claim. The paper does make a good point that under the umbrella term "interpretability" there are varied and conflicting understandings of what, specifically, is meant. That fact has nearly zero relevance to what I'm saying. When I say that people look for mappings from the model to a space of cognitive structures over which they have authority, it could be (and observed in practice always is) a smattering of all the different things talked about in the paper -- authority over which diagnostics represent 'trust,' authority over which visualizations are adequately informative, authority over explaining 'how' an algorithm works, and so on. Though all of these are discordant takes on the specifics of interpretability, as discussed in the paper, they still are unified in the sense that the true intention in appealing to any of them is to map things into a domain where someone wields the authority. It's not about actually caring about trusted diagnostics, or caring about quality visualizations, or caring about communicating the intuition behind a model. Those are just the plausible deniability excuses proffered to try to win the battle over which faction's authority gets center stage. I don't require that anyone have a cross-domain consistent view of the specific definition of interpretability in order for my claims to be a viable explanation, and so it is perfectly compatible with the paper. The paper looks at the on-the-ground specifics used in the arguments over interpretability, meanwhile politics explains the behind-the-scenes motivations of any of those specific choices. The Moral Mazes chapter Dexterity with Symbols is a good place if you want to read on the evidence about how this is carried out in management bureaucracy. It has actually been formally studied in some interesting ways.

moultano 10y ago

> Asking a question such as, "what is it about the data that is causing the model to be fitted like this?" is very healthy and important, but is rarely what anyone means by 'interpretability.' What I see people mean by interpretability is one step back. It's literally just "what is the model actually doing?" in few enough concepts that meat machines can hold them in memory. Sometimes accompanied by a second question "is the thing it is doing dumb or smart?" In my field (search) there are large classes of things that "work" but are dumb or annoying, or just not the product we want to provide, and we have to make product decisions to avoid them. So in that context figuring these things out is really important. >Having said all of this, the most egregious abuses of the nefarious side of 'interpretability' that I have seen have all come in quant finance, which is a situation in which the optimization criteria are literally the identically same thing as the actual objective. You know more about this than me, but it seems like there might be big differences between models in how they respond to unquantifiable downside risk of rare events. How do you deal with that without interpretability? How do you examine the assumptions the model is making that you may not want it to make?