Whilst that's an issue with current popular approaches (e.g. neural networks), I don't think it's too catastrophic. One approach to dealing with this is to use

Whilst that's an issue with current popular approaches (e.g. neural networks), I don't think it's too catastrophic. One approach to dealing with this is to use algorithms which are more amenable to explanations, like decision trees or inductive logic programming: that way, we can see exactly what criteria are being used to reach a decision, and we can use existing regulations/audits/etc. to keep them in check. Another approach is to have the AI system produce evidence rather than an opaque response. One area where this already exists is in automated theorem proving: given a (suitably formalised) mathematical statement, programs like E and Vampire can (try to) determine whether it's true or false. In theory, we could ask such programs whether e.g. a loan application satisfies our eligiblity criteria, but the resulting true/false answer would be about as opaque as using deep learning. However, we don't have to be content with a simple boolean: programs like metis can output proofs of a statement (if it's true), whilst programs like MACE can output specific counterexamples (if it's false). It doesn't seem too far fetched to have an AI system generate justifications for and/or against an application.