Here's another example: an AI, perhaps in a self-driving car, did something that caused harm to people and got the manufacturer hauled into court, and the judge

Here's another example: an AI, perhaps in a self-driving car, did something that caused harm to people and got the manufacturer hauled into court, and the judge issues a court order to "make it stop doing that." How can the manufacturer comply, if they don't understand why the AI makes that decision, and how to change its decision-making process? This isn't just about accountability, it's also about the manufacturers' inability to control the behavior of a black-box AI system. If the manufacturer explains to the judge that they don't really know what's going on inside the AI, the judge won't be inclined to cut them a break.