4 ms·
Watson will go against former best "Jeopardy!" players this Fall. However, I think it's a bit overarching to label this as an application of a natural question
by Jun8 16y ago
Watson will go against former best "Jeopardy!" players this Fall. However, I think it's a bit overarching to label this as an application of a natural question answering system, as is done in this article. "Jeopardy!"'s answer snippets are not like normal questions that, say, a tax software would encounter, I think.
"When one I.B.M. executive suggested taking on “Jeopardy!” he was immediately pooh-poohed. Deep Blue was able to play chess well because the game is perfectly logical, with fairly simple rules; it can be reduced easily to math, which computers handle superbly. But the rules of language are much trickier. At the time, the very best question-answering systems — some created by software firms, some by university researchers — could sort through news articles on their own and answer questions about the content, but they understood only questions stated in very simple language (“What is the capital of Russia?”); in government-run competitions, the top systems answered correctly only about 70 percent of the time, and many were far worse. “Jeopardy!” with its witty, punning questions, seemed beyond their capabilities. What’s more, winning on “Jeopardy!” requires finding an answer in a few seconds. The top question-answering machines often spent longer, even entire minutes, doing the same thing."
Still, it's a stunning achievement. And IBM will definitely recoup all the millions it put into this project from the PR money it has saved. Other big tech companies, take note please.
- jerf 16y ago'"Jeopardy!"'s answer snippets are not like normal questions that, say, a tax software would encounter, I think.' I'm not sure, but are you saying that you think it's easier to answer Jeopardy questions than tax questions? I wouldn't think so. The tighter you constrain the domain, the better the computer will do.
- timwiseman 16y agoBut in some ways, Jeopardy is much more constrained than tax questions. You know that the question will have a relatively short statement, the answer will be in the form of a question with probably no more than 5 words being actually relevant with the rest being used to make it into a question. Furthermore, you know that breadth of knowledge is generally more significant depth. Having a database of every nation's capital and a few significant things about it is most likely more useful than being able to provide an in depth discussion of quantum electrodynamics. Tax questions on the other hand often have lengthy and detailed statements and require an essay style answer. Worse, in some cases, detailed tax advice may actually require judgement and advice. Of course, tax software takes short cuts in this respect. It is not designed to handle complicated situations with nuances. It is designed to handle your average consumer, and even there is constrains the problem by being the one that asks questions and then producing forms rather than answering ad hoc questions. (edit: fixed grammar)
- alextp 16y agoNot only that but you can probably parse a jeopardy "answer" into a series of roughly independent clauses, and then try to predict classes that rank high in these clauses. For example, in the "answer" "This action flick starring Roy Scheider in a high-tech police helicopter was also briefly a TV series" you can get it right just by looking for things that correlate highly with "action flick", "Roy Schneider", "police helicopter", and "TV series".
- alextp 16y agoBut they must surely be doing something fancier than this naive-bayes-style model, otherwise they'd have no use for a roomful of supercomputers.
- zach 16y agoWell, naïve Bayesian inference is supercomputer-level when you use it on a huge universe of data. As Peter Norvig often points out, these kind of tasks are highly data dependent. The supercomputers are probably more used for data access as for raw computation. I can totally imagine Peter writing a forty-line Python app that runs on Google's infrastructure that does about as well.
- Jun8 16y agoYou're right, Jeopardy questions are much more contrained than standard, domain-based questions; however, as the article also points out, these constraints may be very hard for the computer to pick up. In fact, the main reason that Watson is slow compared to humans competing against it is precisely this, that it cannot effectively prune the search space in most cases. Look at how the answer to the Michael Jackson video is generated; the final answer is correct. The runner ups are also relevant but in a very weird sense, surely nothing that a human would come up with.
- joe_the_user 16y agoBut is it something a human brain would come up with and filter out? We don't know because we only know what our minds do, not what our brains do.
- jbarham 16y agoSpeaking as a US resident, I know I'd much prefer to memorize general trivia than the US tax code!
- deleted 16y ago[deleted]
- fragmede 16y agoTax processing software aside, how might this do on the NY times crossword puzzle? On the one hand, the answers are more constrained because they have to fit, but good clues are impossibly cryptic for the average human to parse.
- deleted 16y ago[deleted]