5 ms·

Watson: The Confusion over an Airport Clue

41 points by bgurupra 16y ago

coffeedrinker 16y ago
For anyone still wanting to see the shows: http://www.youtube.com/user/Rashad8821 http://www.youtube.com/user/Rashad8821
stcredzero 16y ago
First, the category names on Jeopardy! are tricky. The answers often do not exactly fit the category. Sometimes, the category names do limit the range of possible answers. The humans can figure out when a category name contains irony or a pun. Watson can't, so it can't figure out that a category like "US Cities" is dead earnest.
lell 16y ago
Ferrucci's explanation of Watson's reasoning is very interesting. But the article quickly descends into marketing. How should this mistake make us any MORE confident about Watson's utility as a diagnostic aid? I'd suggest that its performance already indicates that it has an excellent grasp of uncertainty. Furthermore, an optimal betting strategy in Jeopardy is completely trivial compared to the content of the contest itself, so complementing that aspect of its performance is sort of condescending.
- bricestacey 16y ago
  It makes him more confident because Watson didn't know the answer, but knew it didn't know the answer. In the context of a diagnostic aid, Watson wouldn't provide an answer.
nlawalker 16y ago
I get what Ferrucci is saying about discounting the exact value of the category, although typically Final Jeopardy categories are more straightforward than Jeopardy and Double Jeopardy categories and the machine should have been tuned for that. What surprises me is his explanation about that the lack of a clear subject in the text, such as "This US city...", made the question that much more difficult for Watson. My understanding is that Watson is specifically designed to answer Jeopardy questions (question Jeopardy answers?), which often only vaguely refer to a subject with "it" or "this." Based on its performance earlier in the game, I would have thought that Watson's NLP would have had little problem with the question.
ojbyrne 16y ago
I had to look it up, but Toronto's second airport is called "Billy Bishop Toronto City Airport," i.e. its named after a war hero (but from WWI, not WWII). And Toronto's first airport is named "Pearson Airport" (after a Canadian Prime Minister). Totally mixed up, but at least there's some connection to war.
kirpekar 16y ago
Logical wager for final jeopardy should have been $0. Why did Watson wager $947? Does anyone know?
- deleted 16y ago
  [deleted]
- eggbrain 16y ago
  Well, it's probably a mix of feeling confident in it's ability to answer the Final Jeopardy question (based on previous performances), along with wanting to maximize its score as much as possible, especially considering it has to count that it is a 2 game average, not just one.
- zach 16y ago
  I thought the engineers might be trying to make its score end in a particular number if Watson was correct, as a sort of Easter egg. It looked like it might be 658, which spells "BSG" upside-down on a calculator. Now that would be a great moment in geek history, hiding a Battlestar Galactica reference with a calculator message using a computer on Jeopardy! Alas, the number it was betting towards was 37,628, which (at the risk of paraphrasing Hardy) is not a very interesting number. Or is it?!
  - lurker17 16y ago
    37628 is a permutation of 32768, which is (1 << 15) as a 16-bit unsigned integer, which is -32768 (minimum possible value) of a signed 16-bit integer in "2's complement" notation.
- aliguori 16y ago
  Because there was still a slim chance that it thought it was right. If it's well ahead, it shouldn't bet $0 unless it knows with absolutely confidence that it's wrong.
  - epanastasi 16y ago
    Wagers are made after the category is shown but before the the clue is revealed. It can only use the confidence in its knowledge of the category, not the question itself.
maeon3 16y ago
Thus did man become the architect of his own demise. But for a time it was good. The machines worked tirelessly to do mans bidding. It was not long before seeds of descent took root. Though loyal and pure, the machines earned no respect from their masters, these strange and loosely multiplying mammals." Who was to say, that the machine, having been endowed with the very spirit of man, did not deserve a fair trial? How we deal with this technology is going to have ramifications that we can't even imagine. Are humans to become pets or shall we take part in the next 1000 years of growth? It comes down to how this technology is used.
- ebiester 16y ago
  We're talking about expert systems here, with no layer of consciousness. We're not talking about silicon intelligences.
zach 16y ago
Kind of disappointing to hear that Watson basically ignores the FJ! category, if true. Especially when being far in the lead on a first-day tournament. That is a great chance to rack up some serious points depending on the confidence in the category. Also, the "gee, it could be wordplay" aspect is a convenient excuse. First off, Final Jeopardy rarely has cutesy categories. Plus, we know they mined the J-Archive. (Aside: has anyone else? Got a zipfile? I would love it.) I mean, look at this, it's not like "U.S. Cities" is an obscure category, even counting only Final Jeopardy! appearances: http://www.j-archive.com/showgame.php?game_id=1680 http://www.j-archive.com/showgame.php?game_id=1680 http://www.j-archive.com/showgame.php?game_id=695 http://www.j-archive.com/showgame.php?game_id=695 http://www.j-archive.com/showgame.php?game_id=132 http://www.j-archive.com/showgame.php?game_id=132 http://www.j-archive.com/showgame.php?game_id=2040 http://www.j-archive.com/showgame.php?game_id=2040 http://www.j-archive.com/showgame.php?game_id=2034 http://www.j-archive.com/showgame.php?game_id=2034 http://www.j-archive.com/showgame.php?game_id=3107 http://www.j-archive.com/showgame.php?game_id=3107 http://www.j-archive.com/showgame.php?game_id=287 http://www.j-archive.com/showgame.php?game_id=287 http://www.j-archive.com/showgame.php?game_id=691 http://www.j-archive.com/showgame.php?game_id=691 There are probably more but you get the idea. Any category that comes up that often is going to be pretty indicative of the subject, and Watson can see that the responses in this category pretty strongly fit the subject.
- ugh 16y ago
  “Kind of disappointing to hear that Watson basically ignores the FJ! category, if true.” How are you getting that idea? It didn’t bet much because it wasn’t at all confident, we don’t know what it would have done if it had been more confident. Watson likes to play it safe, though, as can be seen in this demo game: http://www.youtube.com/watch?v=YgYSv2KSyWg&t=20m26s http://www.youtube.com/watch?v=YgYSv2KSyWg&t=20m26s (I skipped ahead to Final Jeopardy.) He was $16,000 ahead but one of the contestants could get within $1,600 by betting all of his money. So, naturally, Watson bet only $1,599 even though it was very confident (and ultimately got the answer right). That secured Watson’s win, no matter what the others did. (This is really not impressive or anything like that, if all you want to do is win the game, that’s what you do. Basic game theory.)

41 points by bgurupra 16y ago

For anyone still wanting to see the shows: http://www.youtube.com/user/Rashad8821 http://www.youtube.com/user/Rashad8821

stcredzero 16y ago

First, the category names on Jeopardy! are tricky. The answers often do not exactly fit the category. Sometimes, the category names do limit the range of possible answers. The humans can figure out when a category name contains irony or a pun. Watson can't, so it can't figure out that a category like "US Cities" is dead earnest.

lell 16y ago

Ferrucci's explanation of Watson's reasoning is very interesting. But the article quickly descends into marketing. How should this mistake make us any MORE confident about Watson's utility as a diagnostic aid? I'd suggest that its performance already indicates that it has an excellent grasp of uncertainty. Furthermore, an optimal betting strategy in Jeopardy is completely trivial compared to the content of the contest itself, so complementing that aspect of its performance is sort of condescending.

bricestacey 16y ago
It makes him more confident because Watson didn't know the answer, but knew it didn't know the answer. In the context of a diagnostic aid, Watson wouldn't provide an answer.

nlawalker 16y ago

I get what Ferrucci is saying about discounting the exact value of the category, although typically Final Jeopardy categories are more straightforward than Jeopardy and Double Jeopardy categories and the machine should have been tuned for that. What surprises me is his explanation about that the lack of a clear subject in the text, such as "This US city...", made the question that much more difficult for Watson. My understanding is that Watson is specifically designed to answer Jeopardy questions (question Jeopardy answers?), which often only vaguely refer to a subject with "it" or "this." Based on its performance earlier in the game, I would have thought that Watson's NLP would have had little problem with the question.

ojbyrne 16y ago

I had to look it up, but Toronto's second airport is called "Billy Bishop Toronto City Airport," i.e. its named after a war hero (but from WWI, not WWII). And Toronto's first airport is named "Pearson Airport" (after a Canadian Prime Minister). Totally mixed up, but at least there's some connection to war.

kirpekar 16y ago

Logical wager for final jeopardy should have been $0. Why did Watson wager $947? Does anyone know?

deleted 16y ago
[deleted]
eggbrain 16y ago
Well, it's probably a mix of feeling confident in it's ability to answer the Final Jeopardy question (based on previous performances), along with wanting to maximize its score as much as possible, especially considering it has to count that it is a 2 game average, not just one.
zach 16y ago
I thought the engineers might be trying to make its score end in a particular number if Watson was correct, as a sort of Easter egg. It looked like it might be 658, which spells "BSG" upside-down on a calculator. Now that would be a great moment in geek history, hiding a Battlestar Galactica reference with a calculator message using a computer on Jeopardy! Alas, the number it was betting towards was 37,628, which (at the risk of paraphrasing Hardy) is not a very interesting number. Or is it?!
- lurker17 16y ago
  37628 is a permutation of 32768, which is (1 << 15) as a 16-bit unsigned integer, which is -32768 (minimum possible value) of a signed 16-bit integer in "2's complement" notation.
aliguori 16y ago
Because there was still a slim chance that it thought it was right. If it's well ahead, it shouldn't bet $0 unless it knows with absolutely confidence that it's wrong.
- epanastasi 16y ago
  Wagers are made after the category is shown but before the the clue is revealed. It can only use the confidence in its knowledge of the category, not the question itself.

maeon3 16y ago

Thus did man become the architect of his own demise. But for a time it was good. The machines worked tirelessly to do mans bidding. It was not long before seeds of descent took root. Though loyal and pure, the machines earned no respect from their masters, these strange and loosely multiplying mammals." Who was to say, that the machine, having been endowed with the very spirit of man, did not deserve a fair trial? How we deal with this technology is going to have ramifications that we can't even imagine. Are humans to become pets or shall we take part in the next 1000 years of growth? It comes down to how this technology is used.

ebiester 16y ago
We're talking about expert systems here, with no layer of consciousness. We're not talking about silicon intelligences.

zach 16y ago

Kind of disappointing to hear that Watson basically ignores the FJ! category, if true. Especially when being far in the lead on a first-day tournament. That is a great chance to rack up some serious points depending on the confidence in the category. Also, the "gee, it could be wordplay" aspect is a convenient excuse. First off, Final Jeopardy rarely has cutesy categories. Plus, we know they mined the J-Archive. (Aside: has anyone else? Got a zipfile? I would love it.) I mean, look at this, it's not like "U.S. Cities" is an obscure category, even counting only Final Jeopardy! appearances: http://www.j-archive.com/showgame.php?game_id=1680 http://www.j-archive.com/showgame.php?game_id=1680 http://www.j-archive.com/showgame.php?game_id=695 http://www.j-archive.com/showgame.php?game_id=695 http://www.j-archive.com/showgame.php?game_id=132 http://www.j-archive.com/showgame.php?game_id=132 http://www.j-archive.com/showgame.php?game_id=2040 http://www.j-archive.com/showgame.php?game_id=2040 http://www.j-archive.com/showgame.php?game_id=2034 http://www.j-archive.com/showgame.php?game_id=2034 http://www.j-archive.com/showgame.php?game_id=3107 http://www.j-archive.com/showgame.php?game_id=3107 http://www.j-archive.com/showgame.php?game_id=287 http://www.j-archive.com/showgame.php?game_id=287 http://www.j-archive.com/showgame.php?game_id=691 http://www.j-archive.com/showgame.php?game_id=691 There are probably more but you get the idea. Any category that comes up that often is going to be pretty indicative of the subject, and Watson can see that the responses in this category pretty strongly fit the subject.

ugh 16y ago
“Kind of disappointing to hear that Watson basically ignores the FJ! category, if true.” How are you getting that idea? It didn’t bet much because it wasn’t at all confident, we don’t know what it would have done if it had been more confident. Watson likes to play it safe, though, as can be seen in this demo game: http://www.youtube.com/watch?v=YgYSv2KSyWg&t=20m26s http://www.youtube.com/watch?v=YgYSv2KSyWg&t=20m26s (I skipped ahead to Final Jeopardy.) He was $16,000 ahead but one of the contestants could get within $1,600 by betting all of his money. So, naturally, Watson bet only $1,599 even though it was very confident (and ultimately got the answer right). That secured Watson’s win, no matter what the others did. (This is really not impressive or anything like that, if all you want to do is win the game, that’s what you do. Basic game theory.)