One reason to want to understand what some "black box" is doing is distribution/dataset shift, especially in medical applications. For example, suppose you're

One reason to want to understand what some "black box" is doing is distribution/dataset shift, especially in medical applications. For example, suppose you're building a neural net to detect early-stage lung cancer on medical imaging, and you test/train it on patients in a small set of hospitals. Often the hospital name is given on the image, and this can be used as a covariate to improve accuracy (due to the hospitals serving different populations with different demographics). But a model that does this may suffer when put into production at other hospitals. Some real-life examples with this flavor are given at the end of these slides: https://mlhcmit.github.io/slides/lecture10.pdf https://mlhcmit.github.io/slides/lecture10.pdf. See also Section 6.3 of this paper for how interpretability can help choose models with superior generalization: https://arxiv.org/abs/1602.04938 https://arxiv.org/abs/1602.04938.