Apart from the usual regulatory angle (zip code in US often proxies for race), generally the need for interpretability goes with risk and frequency of decisions

Apart from the usual regulatory angle (zip code in US often proxies for race), generally the need for interpretability goes with risk and frequency of decisions being made. The risk and decision frequency profile of "tag photos with names in social media" is different vs. "decide which contract worth tens of millions of dollars is better to strike". Both require some form of ML/Statistical inference but the higher the risk, the more the explainability required by the decision maker. One strategy you can adopt is to break up a big decision into lots of smaller decisions (eg. Buy ads individually on auction vs. bulk publisher deals) but that kind of approach often comes with it own other costs (Infra to handle scale, transaction costs, lost negotiation leverage). In any real business investment scenario, you usually end up with many many decisions that are somewhat higher risk and they require explainability.