The question is what's that "something"? Is that "something" what we think it is? This focus on explainability is good, and I liked the article, but I think at

The question is what's that "something"? Is that "something" what we think it is? This focus on explainability is good, and I liked the article, but I think at some point this is all has to devolve into scientific theory in general. That is, you can either analytically characterize the inner workings of your ML model, or you can't. If you can characterize it, you should be able to pinpoint what feature is underling a classification decision. If you can't, it becomes an empirical question and therefore no different from any other black box scientific inquiry. In that case you're left with manipulation of feature sets, observing results, and looking for a statistical relationship between the two; more broadly you have to make inferences from your results and that leads into every sort of issue that is encountered in any other scientific inquiry. I have this hunch that being able to analytically characterize a given ML model must at some level basically be the same as coming up with a closed-form representation of it. If you can do that, you could replace the ML model with something simpler. If you can't, there's probably only a sort of statistical relationship that has no closed-form representation, or is otherwise imperfect in some way.