A neat technique to help 'explain' models is LIME: https://www.oreilly.com/learning/introduction-to-local-interpretable-model-agnostic-explanations-lime https:/

A neat technique to help 'explain' models is LIME: https://www.oreilly.com/learning/introduction-to-local-interpretable-model-agnostic-explanations-lime https://www.oreilly.com/learning/introduction-to-local-inter... There is a video here https://www.youtube.com/watch?v=hUnRCxnydCc https://www.youtube.com/watch?v=hUnRCxnydCc I think this has some better examples than the Panda vs Gibbon example in the OP if you want to 'see' why a model may classify a tree-frog as a tree-frog vs a billiard (for example). IMO this suggests some level of anthropomorphizing is useful for understanding and building models as the pixels the model picks up aren't really too dissimilar to what I imagine a naive, simple, mind might use. (i.e the tree-frog's goofy face) We like to look at faces for lots of reasons but one of them probably is because they're usually more distinct which is the same, rough, reason why the model likes the face. This is interesting (to me at least) even if it's just matrix multiplication (or uncrumpling high dimensional manifolds) underneath the hood,