4 ms·

Randall Monroe shared a nice series of these last year: https://twitter.com/xkcd/status/1333529967079120896?s=20 https://twitter.com/xkcd/status/133352996707912

Randall Monroe shared a nice series of these last year: https://twitter.com/xkcd/status/1333529967079120896?s=20 https://twitter.com/xkcd/status/1333529967079120896?s=20 "What year did Tom Hanks land on the moon?" "1970" "When did the Simpsons learn to control fire?" "Dec 17, 1989" "Which river did George Washington drown in?" "The Allegheny River" It's got to be tough to train an AI that can generalize to (or at least recognize) questions outside its training set. Unfortunately, when 99% of questions containing 'when' and 'Neil Armstrong' have the answer 1969, it will need a pretty sophisticated algo to avoid the trap of learning that it should always answer 1969. I wrote up another example of this last year based on a question with a real answer that Google still gets wrong: https://www.tedsanders.com/why-does-google-think-the-slowest-animal-is-the-sloth/ https://www.tedsanders.com/why-does-google-think-the-slowest... Google (and plenty of other search engines) will all helpfully tell you that the slowest animal is the sloth, which is plainly incorrect if you just look at a video of a sloth and then look at a video of a worm. Despite their reputation, sloths still move visibly.

vlovich123 5y ago
The last one actually is an example of the ambiguity of language. For example, if you asked this from a person, answers like “sponge”, “mollusk” and “flatworm” probably aren’t what you’re really considering and you might think the other person is being too pedantic. So really you want nuance from the AI like “if you meant all animals, X. If you meant all land animals, Y. If you meant mammals, Z”. Human are very ambiguous with categories because all categories are arbitrary human classifications and dealing with that ambiguity is difficult even for humans.
jcranmer 5y ago
A fun one I tried was "Which president became supreme dictator" that answered "William Howard Taft." It took some finessing to get it (I originally started with "dictator for life"), but it's clear that it picked up only "supreme" and "president" in the query to guess that you meant to ask which president became a Supreme Court justice.
II2II 5y ago
The problem isn't so much the answers, but how they are framed. In the case of Neil Armstrong, the failure is to state that the answer is for a Moon landing. That would allow the person making the query to realize that the answer was not what they were looking for. In this case it is because Armstrong never landed on Mars. In other cases it may be because the question or data were incorrect. The current Tom Hanks answer is closer to what should have been done in this regard since it refers to the film directly, though it remains problematic in that it highlights the year to such a degree that the person making the query may ignore the context. (It is also problematic because the film is about a mission that didn't land on the Moon, which could only be determined through further research. Granted, that is more along the lines of your slowest animal example.)
kelnos 5y ago
> Google (and plenty of other search engines) will all helpfully tell you that the slowest animal is the sloth, which is plainly incorrect if you just look at a video of a sloth and then look at a video of a worm. Would your average person consider a worm an "animal". In the scientific sense, sure, pretty much everything living that is not a plant is an animal, but I think most people don't think of worms when they think of animals. You can say, "well that's wrong", but that's kinda missing the point. Categories are somewhat arbitrary, and word usage can differ greatly from the scientific or dictionary definition when we're talking about colloquial usage.

vlovich123 5y ago

The last one actually is an example of the ambiguity of language. For example, if you asked this from a person, answers like “sponge”, “mollusk” and “flatworm” probably aren’t what you’re really considering and you might think the other person is being too pedantic. So really you want nuance from the AI like “if you meant all animals, X. If you meant all land animals, Y. If you meant mammals, Z”. Human are very ambiguous with categories because all categories are arbitrary human classifications and dealing with that ambiguity is difficult even for humans.

jcranmer 5y ago

A fun one I tried was "Which president became supreme dictator" that answered "William Howard Taft." It took some finessing to get it (I originally started with "dictator for life"), but it's clear that it picked up only "supreme" and "president" in the query to guess that you meant to ask which president became a Supreme Court justice.

II2II 5y ago

The problem isn't so much the answers, but how they are framed. In the case of Neil Armstrong, the failure is to state that the answer is for a Moon landing. That would allow the person making the query to realize that the answer was not what they were looking for. In this case it is because Armstrong never landed on Mars. In other cases it may be because the question or data were incorrect. The current Tom Hanks answer is closer to what should have been done in this regard since it refers to the film directly, though it remains problematic in that it highlights the year to such a degree that the person making the query may ignore the context. (It is also problematic because the film is about a mission that didn't land on the Moon, which could only be determined through further research. Granted, that is more along the lines of your slowest animal example.)

kelnos 5y ago

> Google (and plenty of other search engines) will all helpfully tell you that the slowest animal is the sloth, which is plainly incorrect if you just look at a video of a sloth and then look at a video of a worm. Would your average person consider a worm an "animal". In the scientific sense, sure, pretty much everything living that is not a plant is an animal, but I think most people don't think of worms when they think of animals. You can say, "well that's wrong", but that's kinda missing the point. Categories are somewhat arbitrary, and word usage can differ greatly from the scientific or dictionary definition when we're talking about colloquial usage.