I still don’t agree that it’s possible to make the distinction you’re trying to make. For example, let’s say someone starts asking “why?” every time you make a

I still don’t agree that it’s possible to make the distinction you’re trying to make. For example, let’s say someone starts asking “why?” every time you make a statement about the car accident in the court case. If it’s Newtonian physics, eventually we bottom out at questions that require statistical mechanics to answer and it falls apart. The “explainable” answer is unmasked as a fiction that is incompatible with reality. So then you might say from a pragmatic point of view all that matters is that the jurors were happy with the explanation. But then who decides what that metric is? What if Feynman is on the jury? What if a medieval religious leader is on the jury? The standard of “explaining it to people” is not a thing. This just circles back to exactly what I said before, which is that explainability becomes politics. You’re trying to say it’s pragmatic, but who gets to control that standard? Why do banks need “explainable” models? Because of some political fight about who can regulate them. Now you’re trying to convert that arbitrary political goal into some type of formalized standard of machine learning models, which is totally intellectually dishonest and rigged. If a surgeon uses a computer-vision-assisted robot to save lives, does the robot need to be “explainable?” Who decides what that even means? Is it a political standard because of insurance liability? What if a “smart” person can understand the model but a “dumb” person can’t? What if the model deemed more explainable by a political oversight committee also saves fewer lives, thus condemning people to death for the sake of explainability. I just don’t see any way your appeal to just some “reasonable” or “pragmatic” idea of “explaining it to people” can be carried to a logical conclusion that makes any sense.