I've done some experiments with LIME -- it's one of the most promising approaches for extracting prediction reasons from arbitrary and otherwise opaque models.

I've done some experiments with LIME -- it's one of the most promising approaches for extracting prediction reasons from arbitrary and otherwise opaque models. Another interesting approach specific to random forests is decision paths: http://blog.datadive.net/interpreting-random-forests/ http://blog.datadive.net/interpreting-random-forests/ You get some weird outputs from these sometimes though which makes it hard to automatically show them to users. For example, a reason might be "because you liked salad restaurant X you should check out check BBQ place Y" and it's because there happens to be an overlap in the users who like both captured in your model. Yet it can cause cognitive dissonance for the users who are either strictly healthy eaters (salads only, no BBQ) or delude themselves into thinking they are (forgetting how much they actually order BBQ in addition to salads). That's the main challenge I see -- figuring out how to filter out reasons from these approaches that don't jive with common intuitions or, even harder, get people to learn to trust the reasons as counter-intuitive as they may seem.