With very deep models based on Convolution (CNNs, the article is talking about AI on a computer vision problem and CNNs are the standard for these problems) you

With very deep models based on Convolution (CNNs, the article is talking about AI on a computer vision problem and CNNs are the standard for these problems) you loose almost all explainability. There might be ways to partially mitigate that like adding additional "heads" to the model to output some debugging-like data, or run the image in patches and see where the model is more confident about a particular metric of interest. Both these options are not optimal as you still don't see the inner workings of the model in an interpretable way. Transformers models [1], on the other hand, are the de-facto standard for Natural Language Processing/Understanding problems and recent studies [2] show that they can be applied to computer vision tasks. By construction, transformers let you access the so called "attention heads", which carry a good deal of information about what the model believes is important about the input. So that might be a promising route for some explainability in computer vision tasks, but we are quite a bit far away from fully understanding what is going on inside a deep learning model. [1] https://arxiv.org/abs/1706.03762 https://arxiv.org/abs/1706.03762 [2] https://arxiv.org/abs/2010.11929 https://arxiv.org/abs/2010.11929