4 ms·

Linear models with simple coefficients can often be some of the least explainable models, particularly when the assumption of linearity breaks down. [0] is a go

by mlthoughts2018 6y ago

Linear models with simple coefficients can often be some of the least explainable models, particularly when the assumption of linearity breaks down. [0] is a good classic paper on this, demonstrating a simple example where coding error on the inputs leads to erroneous coefficient estimates that are both statistically significant and also of the wrong sign. Meaning, you would believe the coefficients reflect a real relationship between the covariate and the target, and even could claim it’s statistically significant, and yet the actual relationship to the target is of the opposite sign! Any further feature importance scoring based off the coefficient estimates would then become catastrophically misleading. Meanwhile, a model like support vector regression on the same data is capable of automatically handling the non-linearity, at the expense that there’s no more such thing as a coefficient breakdown in the linear space of input features. Does this make it less “explainable”? That would make zero sense. How can it be worse at explaining a data generating mechanism when it is better at predicting that same generating mechanism. What could it possibly mean to explain something you can’t predict or replicate? The field of “explainable” models doesn’t even make the slightest attempt to address this stuff - it just beats up on models that are arbitrarily labeled as “black boxes” (what does that mean?) If a given model X predicts or replicates a data generating process better than Y, then X explains the process better than Y, period. An analogy: Newtonian physics is not “more explainable than” quantum mechanics. Newtonian physics is just more wrong about how the world works than quantum mechanics. [0]: http://www.saramitchell.org/achen04.pdf http://www.saramitchell.org/achen04.pdf

currymj 6y ago
i think "explainable model" literally means "a model you can explain to people". You may have to explain your decision to a judge or customer after the fact, or you might even be asking a layperson to actually compute predictions manually (as is sometimes done in psychology and medicine for things like triage). the question of whether or not the model provides a good explanation for the data-generating process is a distinct one. I think the Achen paper makes a good point that people cannot safely turn linear regression coefficients into stories about the world, although it doesn't seem like they've stopped trying. but assuming you have a way to validate that it makes good predictions (not an unreasonable assumption), an explainable model can be a useful thing to have.
- mlthoughts2018 6y ago
  You’re totally missing the point. “I can explain how this model works” is not something you can claim about a model. You can only claim it about a tuple of (model, assumptions, data, context). In context A, some simple linear regression might be very “explainable.” In context B, that same linear model might be totally not explainable (because the mechanism of coefficients based on the regression’s fitting procedure might be totally incompatible with other details of the situation.) The analogy between Newtonian and quantum mechanics still holds. The fact that you can more easily map Newtonian mechanics onto English words or pictures absolutely doesn’t make it more “explainable.” By that logic, saying “a magic wizard did it” would be the most “explainable” model of all. This is exactly my point. What is “complex” or “simple”? Arbitrary standards of natural language words? The field of explainable models has done no work on this. It starts from some totally arbitrary and confused idea about something both being an accurate model and an explainable model as if they are separate. The closest academic topic to making “explainability” a serious subject would be the philosophy of language and connection to computability theory, like Kolmogorov complexity, PAC learning, VC dimension, Occam’s razor. But these are algorithmic aspects of model complexity in the face of a specific data set, it’s absolutely not some hand wavy “oh but a person could ‘easily’ understand certain verbal acoustic vibrations about this” based on nothing.
  - currymj 6y ago
    i understand your point perfectly well, but I think we've gotten to the crux of the disagreement, at least. I'm thinking about this pragmatically: a model is explainable if you can explain the model to some people you need to explain it to. I mean this in a fuzzy, "arbitrary", natural language sense -- the test of it is to try to give the explanation. I don't think metaphysical questions about the nature of explanation itself are really relevant. Whether this is true definitely does depend on what data, which people, what context, but it's possible to try to make techniques that will be generally useful in many contexts. Getting into the broader issues, I think Newtonian physics is absolutely more explainable than quantum physics. For example, I've seen court cases around car accidents where consultants had to come in and explain basic mechanics (F=ma, friction, etc.) to the jury without reference to any math; and they did an okay job of conveying the essential ideas. By contrast I've never seen a math-free explanation of quantum mechanics that wasn't a complete mess. Of course, "a wizard did it" is even easier to explain, and may be appealing for that reason, but ultimately that model will turn out to be totally useless.

currymj 6y ago

i think "explainable model" literally means "a model you can explain to people". You may have to explain your decision to a judge or customer after the fact, or you might even be asking a layperson to actually compute predictions manually (as is sometimes done in psychology and medicine for things like triage). the question of whether or not the model provides a good explanation for the data-generating process is a distinct one. I think the Achen paper makes a good point that people cannot safely turn linear regression coefficients into stories about the world, although it doesn't seem like they've stopped trying. but assuming you have a way to validate that it makes good predictions (not an unreasonable assumption), an explainable model can be a useful thing to have.

mlthoughts2018 6y ago
You’re totally missing the point. “I can explain how this model works” is not something you can claim about a model. You can only claim it about a tuple of (model, assumptions, data, context). In context A, some simple linear regression might be very “explainable.” In context B, that same linear model might be totally not explainable (because the mechanism of coefficients based on the regression’s fitting procedure might be totally incompatible with other details of the situation.) The analogy between Newtonian and quantum mechanics still holds. The fact that you can more easily map Newtonian mechanics onto English words or pictures absolutely doesn’t make it more “explainable.” By that logic, saying “a magic wizard did it” would be the most “explainable” model of all. This is exactly my point. What is “complex” or “simple”? Arbitrary standards of natural language words? The field of explainable models has done no work on this. It starts from some totally arbitrary and confused idea about something both being an accurate model and an explainable model as if they are separate. The closest academic topic to making “explainability” a serious subject would be the philosophy of language and connection to computability theory, like Kolmogorov complexity, PAC learning, VC dimension, Occam’s razor. But these are algorithmic aspects of model complexity in the face of a specific data set, it’s absolutely not some hand wavy “oh but a person could ‘easily’ understand certain verbal acoustic vibrations about this” based on nothing.
- currymj 6y ago
  i understand your point perfectly well, but I think we've gotten to the crux of the disagreement, at least. I'm thinking about this pragmatically: a model is explainable if you can explain the model to some people you need to explain it to. I mean this in a fuzzy, "arbitrary", natural language sense -- the test of it is to try to give the explanation. I don't think metaphysical questions about the nature of explanation itself are really relevant. Whether this is true definitely does depend on what data, which people, what context, but it's possible to try to make techniques that will be generally useful in many contexts. Getting into the broader issues, I think Newtonian physics is absolutely more explainable than quantum physics. For example, I've seen court cases around car accidents where consultants had to come in and explain basic mechanics (F=ma, friction, etc.) to the jury without reference to any math; and they did an okay job of conveying the essential ideas. By contrast I've never seen a math-free explanation of quantum mechanics that wasn't a complete mess. Of course, "a wizard did it" is even easier to explain, and may be appealing for that reason, but ultimately that model will turn out to be totally useless.