Being able to "explain oneself" is a poor metric. Neural networks are not simulated human reasoning systems, they don't work in a way that is parallel to how we

Being able to "explain oneself" is a poor metric. Neural networks are not simulated human reasoning systems, they don't work in a way that is parallel to how we would see and explain something. Asking for an explanation is asking for a different kind of reasoning system. Every once in a while there is a "no such thing as tacit knowledge" article - to me this is the dual. Anyway, IMO is what's more important is demonstrating an AI system can be trusted for the task it's doing (for example by being clear that the data it's predicting on is similar to what it has encountered before), and using the predictions responsibly (like not making a final high value decision based solely on model output). These things are not trivial and areas of ongoing research. But I think they will be more fruitful for developing useful AI vs trying to "explain" something in a way that will satisfy a human checker. (Also, incidentally, there is work showing that much of explainability - for example feature maps - is just building something that produces output a human wants to see. It doesn't actually or necessarily correspond to how the NN calculated it output)