Not exactly. People can be asked to provide their sources or reasoning, which an informed user can easily check and verify. This is not the case with black-box

Not exactly. People can be asked to provide their sources or reasoning, which an informed user can easily check and verify. This is not the case with black-box models. They only point to an answer/give a reply without any justification or reasoning behind it. This is actually a very severe problem with black-box models: ultimately they cannot be trusted, because it's very hard to verify whether the learned objective function matches the intended objective function (this is called the alignment problem [1].) Optimisers tend to produce Clever Hans instances whenever they can, because it's the cheapest and therefore most optimal solution. Even if this becomes obvious from failure cases (e.g. common misclassification in image recognition systems), it's still not obvious which clues the system used that lead to the misclassification. This is in contrast to a person, who an be queried as to why and how they arrived at their conclusion. [1] https://bdtechtalks.com/2021/01/18/ai-alignment-problem-brian-christian/ https://bdtechtalks.com/2021/01/18/ai-alignment-problem-bria...