> Asking a question such as, "what is it about the data that is causing the model to be fitted like this?" is very healthy and important, but is rarely what any

> Asking a question such as, "what is it about the data that is causing the model to be fitted like this?" is very healthy and important, but is rarely what anyone means by 'interpretability.' What I see people mean by interpretability is one step back. It's literally just "what is the model actually doing?" in few enough concepts that meat machines can hold them in memory. Sometimes accompanied by a second question "is the thing it is doing dumb or smart?" In my field (search) there are large classes of things that "work" but are dumb or annoying, or just not the product we want to provide, and we have to make product decisions to avoid them. So in that context figuring these things out is really important. >Having said all of this, the most egregious abuses of the nefarious side of 'interpretability' that I have seen have all come in quant finance, which is a situation in which the optimization criteria are literally the identically same thing as the actual objective. You know more about this than me, but it seems like there might be big differences between models in how they respond to unquantifiable downside risk of rare events. How do you deal with that without interpretability? How do you examine the assumptions the model is making that you may not want it to make?