Yeah, when people want to see e.g. an image classification model explain the different features it saw in the image and the weights it assigned them in making i

Yeah, when people want to see e.g. an image classification model explain the different features it saw in the image and the weights it assigned them in making it's decision (this is an example in the article), they are asking for something that isn't what the model does. ML models have tacit knowledge in a sense, you can't tractably write down a process for it. That's not to say you can't describe the situations in which a model works.