4 ms·

You’re totally missing the point. “I can explain how this model works” is not something you can claim about a model. You can only claim it about a tuple of (mo

by mlthoughts2018 6y ago

You’re totally missing the point. “I can explain how this model works” is not something you can claim about a model. You can only claim it about a tuple of (model, assumptions, data, context). In context A, some simple linear regression might be very “explainable.” In context B, that same linear model might be totally not explainable (because the mechanism of coefficients based on the regression’s fitting procedure might be totally incompatible with other details of the situation.) The analogy between Newtonian and quantum mechanics still holds. The fact that you can more easily map Newtonian mechanics onto English words or pictures absolutely doesn’t make it more “explainable.” By that logic, saying “a magic wizard did it” would be the most “explainable” model of all. This is exactly my point. What is “complex” or “simple”? Arbitrary standards of natural language words? The field of explainable models has done no work on this. It starts from some totally arbitrary and confused idea about something both being an accurate model and an explainable model as if they are separate. The closest academic topic to making “explainability” a serious subject would be the philosophy of language and connection to computability theory, like Kolmogorov complexity, PAC learning, VC dimension, Occam’s razor. But these are algorithmic aspects of model complexity in the face of a specific data set, it’s absolutely not some hand wavy “oh but a person could ‘easily’ understand certain verbal acoustic vibrations about this” based on nothing.

currymj 6y ago
i understand your point perfectly well, but I think we've gotten to the crux of the disagreement, at least. I'm thinking about this pragmatically: a model is explainable if you can explain the model to some people you need to explain it to. I mean this in a fuzzy, "arbitrary", natural language sense -- the test of it is to try to give the explanation. I don't think metaphysical questions about the nature of explanation itself are really relevant. Whether this is true definitely does depend on what data, which people, what context, but it's possible to try to make techniques that will be generally useful in many contexts. Getting into the broader issues, I think Newtonian physics is absolutely more explainable than quantum physics. For example, I've seen court cases around car accidents where consultants had to come in and explain basic mechanics (F=ma, friction, etc.) to the jury without reference to any math; and they did an okay job of conveying the essential ideas. By contrast I've never seen a math-free explanation of quantum mechanics that wasn't a complete mess. Of course, "a wizard did it" is even easier to explain, and may be appealing for that reason, but ultimately that model will turn out to be totally useless.
- mlthoughts2018 6y ago
  I still don’t agree that it’s possible to make the distinction you’re trying to make. For example, let’s say someone starts asking “why?” every time you make a statement about the car accident in the court case. If it’s Newtonian physics, eventually we bottom out at questions that require statistical mechanics to answer and it falls apart. The “explainable” answer is unmasked as a fiction that is incompatible with reality. So then you might say from a pragmatic point of view all that matters is that the jurors were happy with the explanation. But then who decides what that metric is? What if Feynman is on the jury? What if a medieval religious leader is on the jury? The standard of “explaining it to people” is not a thing. This just circles back to exactly what I said before, which is that explainability becomes politics. You’re trying to say it’s pragmatic, but who gets to control that standard? Why do banks need “explainable” models? Because of some political fight about who can regulate them. Now you’re trying to convert that arbitrary political goal into some type of formalized standard of machine learning models, which is totally intellectually dishonest and rigged. If a surgeon uses a computer-vision-assisted robot to save lives, does the robot need to be “explainable?” Who decides what that even means? Is it a political standard because of insurance liability? What if a “smart” person can understand the model but a “dumb” person can’t? What if the model deemed more explainable by a political oversight committee also saves fewer lives, thus condemning people to death for the sake of explainability. I just don’t see any way your appeal to just some “reasonable” or “pragmatic” idea of “explaining it to people” can be carried to a logical conclusion that makes any sense.

currymj 6y ago

i understand your point perfectly well, but I think we've gotten to the crux of the disagreement, at least. I'm thinking about this pragmatically: a model is explainable if you can explain the model to some people you need to explain it to. I mean this in a fuzzy, "arbitrary", natural language sense -- the test of it is to try to give the explanation. I don't think metaphysical questions about the nature of explanation itself are really relevant. Whether this is true definitely does depend on what data, which people, what context, but it's possible to try to make techniques that will be generally useful in many contexts. Getting into the broader issues, I think Newtonian physics is absolutely more explainable than quantum physics. For example, I've seen court cases around car accidents where consultants had to come in and explain basic mechanics (F=ma, friction, etc.) to the jury without reference to any math; and they did an okay job of conveying the essential ideas. By contrast I've never seen a math-free explanation of quantum mechanics that wasn't a complete mess. Of course, "a wizard did it" is even easier to explain, and may be appealing for that reason, but ultimately that model will turn out to be totally useless.

mlthoughts2018 6y ago
I still don’t agree that it’s possible to make the distinction you’re trying to make. For example, let’s say someone starts asking “why?” every time you make a statement about the car accident in the court case. If it’s Newtonian physics, eventually we bottom out at questions that require statistical mechanics to answer and it falls apart. The “explainable” answer is unmasked as a fiction that is incompatible with reality. So then you might say from a pragmatic point of view all that matters is that the jurors were happy with the explanation. But then who decides what that metric is? What if Feynman is on the jury? What if a medieval religious leader is on the jury? The standard of “explaining it to people” is not a thing. This just circles back to exactly what I said before, which is that explainability becomes politics. You’re trying to say it’s pragmatic, but who gets to control that standard? Why do banks need “explainable” models? Because of some political fight about who can regulate them. Now you’re trying to convert that arbitrary political goal into some type of formalized standard of machine learning models, which is totally intellectually dishonest and rigged. If a surgeon uses a computer-vision-assisted robot to save lives, does the robot need to be “explainable?” Who decides what that even means? Is it a political standard because of insurance liability? What if a “smart” person can understand the model but a “dumb” person can’t? What if the model deemed more explainable by a political oversight committee also saves fewer lives, thus condemning people to death for the sake of explainability. I just don’t see any way your appeal to just some “reasonable” or “pragmatic” idea of “explaining it to people” can be carried to a logical conclusion that makes any sense.