3 ms·

I think what people are looking for in ‘explainable AI’ is: when the AI makes a bad decision, they want to be able to look into the neural network and say ‘ther

I think what people are looking for in ‘explainable AI’ is: when the AI makes a bad decision, they want to be able to look into the neural network and say ‘there: that neuron being set to that value is what made the AI mistake the cyclist for a drop kerb’. Then we can fix the value and the AI will not make that mistake again. But when an AI gets sufficiently complex of course there won’t be explanations that make sense for those kinds of errors, because just like a human the AI is integrating lots of different bits of information that it has learned are important and it has limited capacity and sometimes it just gets its attention focused on the wrong thing and it just didn’t see the guy, okay? Demanding that AI be explainable is fundamentally demanding that it not be intelligent.

YeGoblynQueenne 5y ago
Just to be clear, a neural network is not "an AI". "AI" is the name of the research field. We don't have "AIs" as in Science Fiction yet, and neural networks don't do anything "just like a human". When people in AI research talk about "attention" in neural networks, that's just an anthropomorphic, and quite unfortunate, name for a specific technique in training neural nets. It doesn't mean that a machine vision system has the ability to focus its attention in the same way that humans do. That out of the way, there are AI approaches that can explain their actions just fine without going dumb. For example, I posted this comment earlier: https://news.ycombinator.com/item?id=30872400 https://news.ycombinator.com/item?id=30872400 about an AI system called Nook that recently won a tournament against 8 human champions of the card game Bridge. In Bridge, players must be able to explain their moves, so an AI player without the ability to explain its decisions can't play a full game of Bridge.

YeGoblynQueenne 5y ago

Just to be clear, a neural network is not "an AI". "AI" is the name of the research field. We don't have "AIs" as in Science Fiction yet, and neural networks don't do anything "just like a human". When people in AI research talk about "attention" in neural networks, that's just an anthropomorphic, and quite unfortunate, name for a specific technique in training neural nets. It doesn't mean that a machine vision system has the ability to focus its attention in the same way that humans do. That out of the way, there are AI approaches that can explain their actions just fine without going dumb. For example, I posted this comment earlier: https://news.ycombinator.com/item?id=30872400 https://news.ycombinator.com/item?id=30872400 about an AI system called Nook that recently won a tournament against 8 human champions of the card game Bridge. In Bridge, players must be able to explain their moves, so an AI player without the ability to explain its decisions can't play a full game of Bridge.