There's a whole field studying machine learning interpretability. This recent Distill article is a good introduction [1]. The lime [2] package, based on this

There's a whole field studying machine learning interpretability. This recent Distill article is a good introduction [1]. The lime [2] package, based on this paper [3], is also notable. [1]: https://distill.pub/2018/building-blocks/ https://distill.pub/2018/building-blocks/ [2]: https://github.com/marcotcr/lime https://github.com/marcotcr/lime [3]: https://arxiv.org/abs/1602.04938 https://arxiv.org/abs/1602.04938