5 ms·

David Ferrucci, the manager of the Watson project at IBM, on why he thinks Watson got the Final Jeopardy question wrong: "First, the category names on Jeopardy

David Ferrucci, the manager of the Watson project at IBM, on why he thinks Watson got the Final Jeopardy question wrong: "First, the category names on Jeopardy! are tricky. The answers often do not exactly fit the category. Watson, in his training phase, learned that categories only weakly suggest the kind of answer that is expected, and, therefore, the machine downgrades their significance. The way the language was parsed provided an advantage for the humans and a disadvantage for Watson, as well. “What US city” wasn’t in the question. If it had been, Watson would have given US cities much more weight as it searched for the answer. Adding to the confusion for Watson, there are cities named Toronto in the United States and the Toronto in Canada has an American League baseball team. It probably picked up those facts from the written material it has digested. Also, the machine didn’t find much evidence to connect either city’s airport to World War II. (Chicago was a very close second on Watson’s list of possible answers.) So this is just one of those situations that’s a snap for a reasonably knowledgeable human but a true brain teaser for the machine." http://asmarterplanet.com/blog/2011/02/watson-on-jeopardy-day-two-the-confusion-over-an-airport-clue.html http://asmarterplanet.com/blog/2011/02/watson-on-jeopardy-da...

gojomo 16y ago
Lame excuses! Watson is impressive, but I'm disappointed by the lack of any hint of comprehension behind its answers. When it's wrong, it's often nonsensically out-to-lunch, and its 2nd/3rd best answers are also often batty. If it's just trained-up on statistical correlations between trigger phrases and likely answers in the constrained Jeopardy domain, then 90 32-core/512GB RAM servers seem like overkill.
- InclinedPlane 16y ago
  Watson also generates confidence estimates and minimum confidence bars for questions. It may sometimes have batty "answers" but usually it knows they are batty. The rate of incorrect answers that Watson has a high confidence in is fairly low. What's remarkable and important to not take lightly is the result that it's possible to generate answers to often vague and indirect clues without understanding. That likely means that it will be possible to build useful systems for automating research and the synthesis of large amounts of data without needing to build artificial human-level intelligence.
  - extension 16y ago
    ...presuming that human-level intelligence entails any sort of understanding that is fundamentally deeper than what Watson is doing.
    - tel 16y ago
      I think that's a fair bet. The new wave of "probablistic everywhere" NLP models, though even the very simplest strictly dominate older grammatical methods, are not often capable of taking advantage of a lot of the structure of language and topic that humans are wont to do. It's a cutting-edge accomplishment when NLP algorithms learn prediction of long-range word pairs such as how you almost certainly will see "law" or "marriage" somewhere in a sentence containing the word "annulled" even if that local area of the sentence doesn't seem to call for it. Humans on the other hand are more likely to forget that it's possible to annul pretty much anything else. I don't own a TV and plan on watching the Jeopardy match later online, so I'm just going to guess about Watson's performance. I think that humans abuse discovered patterns and structure in language and meaning to search through possible interpretations very quickly. Watson on the other hand uses far less structure and a room full of 200 cores to search through everything is knows much less efficiently. I feel like Watson's "strange" answers probably aren't nearly so strange when you realize it's simply being more fair to any possible answer than a human would. What's scart is this sort of thing---a willingness to consider out of context answers---sounds pretty similar to the kind of behaviors we humans praise as creative!
      - extension 16y ago
        I think that humans abuse discovered patterns and structure in language and meaning to search through possible interpretations very quickly. Right, but does that structure really represent a "deeper" understanding or just vast and meticulous optimizations of statistical algorithms similar to Watson's? Or is there a difference? We feel like we know how we think, but we can't actually explain it in enough detail to reproduce. Humans have a bad history of rationalization and tunnel vision. And now we discover that all the "wrong" ways to think deeply are actually the right ways to make a working AI. If the AI can fool us into believing that it "understands" then maybe we can fool ourselves in the same way.
- bl4k 16y ago
  If you forced a person to give four answers to a question and rank them in probability of being correct, then at least 3 and possibly 4 of them will look 'batty'
  - gojomo 16y ago
    Not at all! 'Batty' means even a person who doesn't know the right answer can tell they're wrong. Even very bad people at Jeopardy¹ usually draw candidate answers from a more plausible set than is shown in Watson's logic/errors. The best case I can make for Watson is that perhaps the alternatives shown are each actually the top option based on totally different understandings of the question. So in fact many other plausible answers are folded up just behind its right answer. The shown alternatives are meant to be: if the question means something else entirely. ¹ eg Wolf Blitzer: http://www.youtube.com/watch?v=DVC28oemocA http://www.youtube.com/watch?v=DVC28oemocA

gojomo 16y ago

Lame excuses! Watson is impressive, but I'm disappointed by the lack of any hint of comprehension behind its answers. When it's wrong, it's often nonsensically out-to-lunch, and its 2nd/3rd best answers are also often batty. If it's just trained-up on statistical correlations between trigger phrases and likely answers in the constrained Jeopardy domain, then 90 32-core/512GB RAM servers seem like overkill.

InclinedPlane 16y ago
Watson also generates confidence estimates and minimum confidence bars for questions. It may sometimes have batty "answers" but usually it knows they are batty. The rate of incorrect answers that Watson has a high confidence in is fairly low. What's remarkable and important to not take lightly is the result that it's possible to generate answers to often vague and indirect clues without understanding. That likely means that it will be possible to build useful systems for automating research and the synthesis of large amounts of data without needing to build artificial human-level intelligence.
- extension 16y ago
  ...presuming that human-level intelligence entails any sort of understanding that is fundamentally deeper than what Watson is doing.
  - tel 16y ago
    I think that's a fair bet. The new wave of "probablistic everywhere" NLP models, though even the very simplest strictly dominate older grammatical methods, are not often capable of taking advantage of a lot of the structure of language and topic that humans are wont to do. It's a cutting-edge accomplishment when NLP algorithms learn prediction of long-range word pairs such as how you almost certainly will see "law" or "marriage" somewhere in a sentence containing the word "annulled" even if that local area of the sentence doesn't seem to call for it. Humans on the other hand are more likely to forget that it's possible to annul pretty much anything else. I don't own a TV and plan on watching the Jeopardy match later online, so I'm just going to guess about Watson's performance. I think that humans abuse discovered patterns and structure in language and meaning to search through possible interpretations very quickly. Watson on the other hand uses far less structure and a room full of 200 cores to search through everything is knows much less efficiently. I feel like Watson's "strange" answers probably aren't nearly so strange when you realize it's simply being more fair to any possible answer than a human would. What's scart is this sort of thing---a willingness to consider out of context answers---sounds pretty similar to the kind of behaviors we humans praise as creative!
    - extension 16y ago
      I think that humans abuse discovered patterns and structure in language and meaning to search through possible interpretations very quickly. Right, but does that structure really represent a "deeper" understanding or just vast and meticulous optimizations of statistical algorithms similar to Watson's? Or is there a difference? We feel like we know how we think, but we can't actually explain it in enough detail to reproduce. Humans have a bad history of rationalization and tunnel vision. And now we discover that all the "wrong" ways to think deeply are actually the right ways to make a working AI. If the AI can fool us into believing that it "understands" then maybe we can fool ourselves in the same way.
bl4k 16y ago
If you forced a person to give four answers to a question and rank them in probability of being correct, then at least 3 and possibly 4 of them will look 'batty'
- gojomo 16y ago
  Not at all! 'Batty' means even a person who doesn't know the right answer can tell they're wrong. Even very bad people at Jeopardy¹ usually draw candidate answers from a more plausible set than is shown in Watson's logic/errors. The best case I can make for Watson is that perhaps the alternatives shown are each actually the top option based on totally different understandings of the question. So in fact many other plausible answers are folded up just behind its right answer. The shown alternatives are meant to be: if the question means something else entirely. ¹ eg Wolf Blitzer: http://www.youtube.com/watch?v=DVC28oemocA http://www.youtube.com/watch?v=DVC28oemocA