Depending on the algorithm, there is this: https://homes.cs.washington.edu/~marcotcr/blog/lime/ https://homes.cs.washington.edu/~marcotcr/blog/lime/ It's a nea

Depending on the algorithm, there is this: https://homes.cs.washington.edu/~marcotcr/blog/lime/ https://homes.cs.washington.edu/~marcotcr/blog/lime/ It's a neat approach that I've used with some random forest classifiers.