So this is pretty limited. It only works on text data, and just picks the part of the text that most determines the output. The basic idea of doing this isn't n

So this is pretty limited. It only works on text data, and just picks the part of the text that most determines the output. The basic idea of doing this isn't new. You can easily ask a naive bayes spam filter what words it thinks are the highest evidence of spam or not spam in a document. It is interesting to see this done with neural nets though. I recall reading something similar that uses gradients to find what words changed the network's output the most, I'm not sure if this new method is actually better and it's not as general. But this is a long way away from the NN being able to give understandable reasons for its decisions. These methods will always be limited to pointing to a part of the input and saying "that part seemed relevant". But it can never articulate why it's relevant, or what it's "thinking" internally. I think this is ok though. I mean looking at what features the model is using to make predictions is pretty useful and should give you a rough idea how it works. I've wondered in the past if neural networks could train humans to understand them. The human would be shown an input, and try to predict the value of a neuron in the network. So the human would learn what the network has learned, and gain intuition about the inner workings of the model. You can also do a similar process with other machine learning methods. You can train a simpler, more understandable model, like decision trees, to predict the neurons of a neural net. And then the human can study that. You can even train a smaller neural network to fit to a bigger, more complex one.