You can look at the gradients into the neural network and see what features are changing the output and by how much. This approach is called LIME, see https://g

You can look at the gradients into the neural network and see what features are changing the output and by how much. This approach is called LIME, see https://github.com/marcotcr/lime https://github.com/marcotcr/lime