I've thought a bit about this problem. I definitely see the reasoning behind it, especially in industries like Healthcare, or anything public-health related (l

I've thought a bit about this problem. I definitely see the reasoning behind it, especially in industries like Healthcare, or anything public-health related (like self-driving cars) etc. It doesn't seem like it's as big a problem as I originally thought. If I understand correctly: At least with models that categorize data, wouldn't the internals of these models be able to determine where the correlations are being formed between neurons? Based on these "strong links" a model should probably be able to hook in some "analysis system" which will take the LABELS of each of the original inputs and explain that the model finds that a correlation between X and Y tends to indicate a particular output. I'm over-simplifying a bit, but in general this seems like it could be converted to natural language relatively easily. However, this over-simplification is perhaps the new problem in that the explanations could end up being extremely long depending on how complex the problem they're trying to solve is. Perhaps, instead, a side-by-side comparison of a "visual graph" of the model's "theoretical ideal representation" of an idea vs what particular inputs produced. This idea could be explored further by allowing experts in a field to confirm if the correlations being represented in a model are in fact correlations that scientists also believe exist based on their own studies. In other words, model A says a correlation exists between the person's first name starting with 'X' and the amount of oxygen in that person's blood cells. Obviously this model would be abandoned pretty quickly :) Also I was recently thinking about how a lot of times, since individual models will piece together different correlations at different strengths, by averaging the outputs of many models (ensemble learning), it's almost as if you're polling a quorum of individuals for their answers. Like that tv show "Who wants to be a millionaire" individuals within an audience may not know the answer, but overall the audience (as a whole) almost always gets the answer right. So the explanation would look more like "the majority of the models have concluded that a correlation exists between X and Y. The data in question exhibits this correlation".