The DARPA proposal the authors are working on is driven by the military. Consider the applications of an AI whose job it is to find hidden weapons caches from a

The DARPA proposal the authors are working on is driven by the military. Consider the applications of an AI whose job it is to find hidden weapons caches from aerial photography of a city, or likely location for IEDs along a road. You could train a NN to do either, given lots of training data, but before risking the lives of special forces, and potential for collateral damage, you might want to question the AI as to why they think what they found was a hidden cache of weapons or roadside IED. I agree the statement "humans are surprisingly good at explaining their decisions" is dubious at best. Most explanations are a matter of what's called reconstructive cognition, which is far from accurate in explaining the reasons for an action.