A key reason that we want models at least for some applications to be interpretable is to watch out for undesirable features. For example, suppose we want to tr

A key reason that we want models at least for some applications to be interpretable is to watch out for undesirable features. For example, suppose we want to train a model to figure out whether to grant or deny a loan, and we train it to match the decisions of human loan officers. Now, suppose it turns out that many loan officers have unconscious prejudices that cause them to deny loans more often to green people and grant loans more often to blue people (substitute whatever categories you like for blue/green). The model might wind up with an explicit weight that makes this implicit discrimination explicit. If the model is relatively small and interpretable this weight can be found and perhaps eliminated. But if that model could chat with us it would replicate the speech of the loan officers, many of whom sincerely believe that they treat green people and blue people fairly. So interpretability can't be about somehow asking the model to justify itself. We may need the equivalent of a debugger.