2 ms·

You could make something like an "expert system" with it. But since it doesn't seem to link back to the source, you'd never know if it was giving you a real ans

• whimsicalism 6y ago
  A bit like people
  • qayxc 6y ago
    Not exactly. People can be asked to provide their sources or reasoning, which an informed user can easily check and verify. This is not the case with black-box models. They only point to an answer/give a reply without any justification or reasoning behind it. This is actually a very severe problem with black-box models: ultimately they cannot be trusted, because it's very hard to verify whether the learned objective function matches the intended objective function (this is called the alignment problem [1].) Optimisers tend to produce Clever Hans instances whenever they can, because it's the cheapest and therefore most optimal solution. Even if this becomes obvious from failure cases (e.g. common misclassification in image recognition systems), it's still not obvious which clues the system used that lead to the misclassification. This is in contrast to a person, who an be queried as to why and how they arrived at their conclusion. [1] https://bdtechtalks.com/2021/01/18/ai-alignment-problem-brian-christian/ https://bdtechtalks.com/2021/01/18/ai-alignment-problem-bria...