10 ms·

Ken Jennings on Watson

22 points by amichail 16y ago

blhack 16y ago
In summary: the buzzer. You could pretty clearly see this if you were watching the game. Ken looked pretty bummed out every time he clearly knew the answer, but missed buzzing in by a few milliseconds or so. Out of curiosity, how does Jeopardy work? Can you buzz in as soon as you know the answer, or do you have to wait for the host to finish?
- deleted 16y ago
  [deleted]
- martythemaniak 16y ago
  Once Trebek finishes, the buzzers start working.
- rst 16y ago
  You have to wait for the host to finish. Or, more precisely, you have to wait for the guy backstage who pushes his own button after the host has finished. If you push your button before he pushes his, you're locked out for something like half a second. His button is also connected to some light bulbs which are never shown on camera, which tell the contestants what's going on. (But there are some human contestants who ignore the lights and try to time it anyway --- having practiced a lot at home, where they couldn't see them. One of them was a recent college champion, who was one of the pre-show speakers at a group viewing I was at for Monday's show.)
- adestefan 16y ago
  Watson received an electronic signal that the buzzers were active. The human contestants have to time it via Trebek's voice or the light.
  - endtime 16y ago
    An electronic signal and a photonic signal ("the light") both travel at the same speed.
    - orborde 16y ago
      An incandescent light might take a nontrivial amount of time (tenth of a second?) to heat up and light after power is applied.
    - locopati 16y ago
      But a computer's detection and response to a signal is probably faster than a human's reaction time.
zck 16y ago
I'd be interested to know if Jennings had any influence on the title -- the phrase "unfair advantage" isn't used in the text -- he calls it an "advantage", but "unfair" lends it an entirely different meaning. The article is a very even-handed discussion of Watson; the headline is needlessly provocative, and makes Jennings sound like a whiner.
- sciolistse 16y ago
  In his Q&A session after day 1, Jennings said that "I wouldn't call this unfair...precise timing just happens to be one thing computers are better at than we humans. It's not like I think Watson should try buzzing in more erratically just to give homo sapiens a chance." So my guess is that the daily news had more to do with that title than Ken. It doesn't seem like him, at any rate.
  - m_myers 16y ago
    Judging from the other content advertised on the page, the Daily News is not a master of subtlety.
- raheemm 16y ago
  I'm certain it was a copywriter who created that headline. Would you read "How to learn to play Piano" or "They Laughed Until I sat down to play?" I know HN tends to dislike overtly dramatic headlines but copywriting is also an interesting art.
tejaswiy 16y ago
What in the world. The challenge was the natural language processing. Considering that Watson did that part admirably well, complaining about computers having better reflexes is just silly.
- tezmc 16y ago
  I seem to recall Kasparov complaining of unfairness/cheating too, when he was beat by Deep Blue.
- adestefan 16y ago
  The buzzer timing is a larger part of winning than knowing the answers. I would say, even though Jennings never does, that it was a huge advantage.
  - kenjackson 16y ago
    But the buzzer timing is also part of Jeopardy. It's just a part of the game. You can change the game to disadvantage the computer, but then why not change other rules like require all responses to be handwritten using a hand with human blood flowing through it to a pumping heart, which is connected to a working human brain.
- powrtoch 16y ago
  The point is that while Watson is a big step forward in language processing and information retrieval, it's unfair to conclude that he does these things as well as the best humans. Jeopardy isn't a particularly good test of these abilities (though it is good theater) because the additional variable of buzzer-timing screws up your results. As was pointed out by another commenter, if the game had been "everyone who knows the answer write it down and get credit", Watson probably wouldn't have won, and absolutely not by so much. So I think Jennings is just cautioning us not to assume that AI has achieved human level language and information capabilities when really a totally unrelated and unimpressive other dimension of the game is the culprit.
jhamburger 16y ago
They should have had a human from the IBM team manning the buzzer based on a visual cue that Watson is ready to answer. Much like Deep Blue didn't actually move the chess pieces with a robotic arm.
- tghw 16y ago
  Why? In this case, Watson use a buzzer just like the human contestants. It still has to take time to figure out the answer.
- marshray 16y ago
  Or to give Watson a taste of the human world perhaps they could route the signal via a congested cell carrier data plan.
- blhack 16y ago
  This is a great idea, actually. This was supposed to be a demonstration of natural language parsing and information retrieval, not a demonstration of a computers ability to react quickly. We already know that computers are good at the latter.
moskie 16y ago
This makes we wish we could rate Watson's performance against the players in a way that didn't involve the buzzer. For instance, if Watson and each player were able to give responses for every single clue in the game, who would have the highest percentage of correct responses and/or the highest score?
- kenjackson 16y ago
  Except, that's not Jeopardy. That's "Answer these Trivia Questions". A legitimate game, but not one that is likely to get much viewership. Everyone misses what should be knocked about Watson. The advantage isn't the buzzer per se. But rather the text message to get the question and the signal to indicate the question had been read. The way it should work in the future is that Watson reads the question from the screen, listens to Alex, and sees the signal from the light. Given the standard way Jeopardy text is written though, I suspect that all of these additional requirements won't change the outcome by much. Watson is just a better Jeopardy player than humans. Are there other trivia games where humans are better? Probably so. But people don't watch these other games in prime time.
  - moskie 16y ago
    Well, right, that wouldn't be Jeopardy, but ultimately, I don't really care about Jeopardy, thus my question. What I (and most of us here, I think) would find interesting is how Watson's language processing abilities compares to that of (very smart) people. I don't really care how fast Watson can ring a buzzer. If there was a way to gauge that, I'd find it interesting.
    - kenjackson 16y ago
      I assume you mean "natural" language processing. For that trivia is the wrong test. It's better to just give it paragraphs and ask it questions about the paragraph. And of course have increasing levels of difficult. More like the SAT reading comprehension test. This would extract how it stores data brought in from structured/unstructed sources and the actual task of processing natural language. I suspect it will be a fair bit worse at processing natural language than even the average person.
warmfuzzykitten 16y ago
Died with his buzzer in his hand, Lord Lord... Nice image by Jennings. But I was surprised to hear him complain about the buzzer reaction time advantage. In an earlier interview with the Washington Post he discounted this, saying that the best human players anticipate the timing while Watson has to wait for the signal. "On any given night, nearly all the contestants know nearly all the answers, so it's just a matter of who masters buzzer rhythm the best... But I wouldn't call this unfair...precise timing just happens to be one thing computers are better at than we humans. It's not like I think Watson should try buzzing in more erratically just to give homo sapiens a chance." http://live.washingtonpost.com/jeopardy-ken-jennings.html?hpid=talkbox1 http://live.washingtonpost.com/jeopardy-ken-jennings.html?hp...
- sausagefeet 16y ago
  Like sciolistse said I think the "unfair" part was a bit of editorializing.
protomyth 16y ago
I'd be more impressed by this massive IBM advertisement, if the computer received its information the exact way the other contestants did (verbally / visually looking at screen).
- ddlatham 16y ago
  It wouldn't make much difference. OCR on the text on the screen would be fine. If it had to do speech recognition for Alex Trebek, that would make it more difficult.
  - protomyth 16y ago
    Yep, it would be more difficult and probably make its buzzer reaction time a little more "honest".
    - bryanlarsen 16y ago
      More honest? It would make it impossible, since if the computer had to parse Trebek's voice, it would finish getting the entire clue at the same time that the humans are pressing the buzzer.
    - kooshball 16y ago
      the OCR would be nearly perfect since there's very little variation in the way the text is displayed. The voice part wouldn't matter since the all top level players use the visual clue to determine the answer long before Trebek finishes reading it.
- jessedhillon 16y ago
  Are you the same guy who left this comment: http://www.crunchgear.com/2011/02/14/angry-nerds-ibms-watson-finishes-night-one-of-jeopardy-challenge-tied-for-first-place/#comment-146803262 http://www.crunchgear.com/2011/02/14/angry-nerds-ibms-watson...
  - protomyth 16y ago
sosuke 16y ago
I think I remember reading that Watson didn't buzz in until it was sure it had an answer which was giving the human opponents an advantage because they could buzz in based of thinking that by the time they are asked for their answer they can have come up with the correct one. Looks like they must have overcome that issue.
deleted 16y ago
[deleted]
hyko 16y ago
John Henry beat the machine he was up against: http://en.wikipedia.org/wiki/John_Henry_(folklore) http://en.wikipedia.org/wiki/John_Henry_(folklore)
CWuestefeld 16y ago
From a competition perspective, I think it was fair: every player is entitled to capitalize his own advantages within the rules. However, I think this would have been more interesting intellectually to isolate Watson's "thinking", with everything else on a level playing field.
ajscherer 16y ago
His analogy about Kasparov needing to solve a long division problem is whack. Buzzing is part of the game of Jeapordy. For all we know he wouldn't have been a champion in a version of the game that didn't involve buzzing.
ConceptDog 16y ago
Ugh. He should have skipped writing about this. No matter the validity of his words, whatever he says will be percieved as sour grapes.
juddlyon 16y ago
No use in complaining, this was a cool PR stunt. You don't hear the Washington Generals bitching about the Globetrotters.

22 points by amichail 16y ago

In summary: the buzzer. You could pretty clearly see this if you were watching the game. Ken looked pretty bummed out every time he clearly knew the answer, but missed buzzing in by a few milliseconds or so. Out of curiosity, how does Jeopardy work? Can you buzz in as soon as you know the answer, or do you have to wait for the host to finish?

deleted 16y ago
[deleted]
martythemaniak 16y ago
Once Trebek finishes, the buzzers start working.
rst 16y ago
You have to wait for the host to finish. Or, more precisely, you have to wait for the guy backstage who pushes his own button after the host has finished. If you push your button before he pushes his, you're locked out for something like half a second. His button is also connected to some light bulbs which are never shown on camera, which tell the contestants what's going on. (But there are some human contestants who ignore the lights and try to time it anyway --- having practiced a lot at home, where they couldn't see them. One of them was a recent college champion, who was one of the pre-show speakers at a group viewing I was at for Monday's show.)
adestefan 16y ago
Watson received an electronic signal that the buzzers were active. The human contestants have to time it via Trebek's voice or the light.
- endtime 16y ago
  An electronic signal and a photonic signal ("the light") both travel at the same speed.
  - orborde 16y ago
    An incandescent light might take a nontrivial amount of time (tenth of a second?) to heat up and light after power is applied.
  - locopati 16y ago
    But a computer's detection and response to a signal is probably faster than a human's reaction time.

zck 16y ago

I'd be interested to know if Jennings had any influence on the title -- the phrase "unfair advantage" isn't used in the text -- he calls it an "advantage", but "unfair" lends it an entirely different meaning. The article is a very even-handed discussion of Watson; the headline is needlessly provocative, and makes Jennings sound like a whiner.

sciolistse 16y ago
In his Q&A session after day 1, Jennings said that "I wouldn't call this unfair...precise timing just happens to be one thing computers are better at than we humans. It's not like I think Watson should try buzzing in more erratically just to give homo sapiens a chance." So my guess is that the daily news had more to do with that title than Ken. It doesn't seem like him, at any rate.
- m_myers 16y ago
  Judging from the other content advertised on the page, the Daily News is not a master of subtlety.
raheemm 16y ago
I'm certain it was a copywriter who created that headline. Would you read "How to learn to play Piano" or "They Laughed Until I sat down to play?" I know HN tends to dislike overtly dramatic headlines but copywriting is also an interesting art.

tejaswiy 16y ago

What in the world. The challenge was the natural language processing. Considering that Watson did that part admirably well, complaining about computers having better reflexes is just silly.

tezmc 16y ago
I seem to recall Kasparov complaining of unfairness/cheating too, when he was beat by Deep Blue.
adestefan 16y ago
The buzzer timing is a larger part of winning than knowing the answers. I would say, even though Jennings never does, that it was a huge advantage.
- kenjackson 16y ago
  But the buzzer timing is also part of Jeopardy. It's just a part of the game. You can change the game to disadvantage the computer, but then why not change other rules like require all responses to be handwritten using a hand with human blood flowing through it to a pumping heart, which is connected to a working human brain.
powrtoch 16y ago
The point is that while Watson is a big step forward in language processing and information retrieval, it's unfair to conclude that he does these things as well as the best humans. Jeopardy isn't a particularly good test of these abilities (though it is good theater) because the additional variable of buzzer-timing screws up your results. As was pointed out by another commenter, if the game had been "everyone who knows the answer write it down and get credit", Watson probably wouldn't have won, and absolutely not by so much. So I think Jennings is just cautioning us not to assume that AI has achieved human level language and information capabilities when really a totally unrelated and unimpressive other dimension of the game is the culprit.

jhamburger 16y ago

They should have had a human from the IBM team manning the buzzer based on a visual cue that Watson is ready to answer. Much like Deep Blue didn't actually move the chess pieces with a robotic arm.

tghw 16y ago
Why? In this case, Watson use a buzzer just like the human contestants. It still has to take time to figure out the answer.
marshray 16y ago
Or to give Watson a taste of the human world perhaps they could route the signal via a congested cell carrier data plan.
blhack 16y ago
This is a great idea, actually. This was supposed to be a demonstration of natural language parsing and information retrieval, not a demonstration of a computers ability to react quickly. We already know that computers are good at the latter.

moskie 16y ago

This makes we wish we could rate Watson's performance against the players in a way that didn't involve the buzzer. For instance, if Watson and each player were able to give responses for every single clue in the game, who would have the highest percentage of correct responses and/or the highest score?

kenjackson 16y ago
Except, that's not Jeopardy. That's "Answer these Trivia Questions". A legitimate game, but not one that is likely to get much viewership. Everyone misses what should be knocked about Watson. The advantage isn't the buzzer per se. But rather the text message to get the question and the signal to indicate the question had been read. The way it should work in the future is that Watson reads the question from the screen, listens to Alex, and sees the signal from the light. Given the standard way Jeopardy text is written though, I suspect that all of these additional requirements won't change the outcome by much. Watson is just a better Jeopardy player than humans. Are there other trivia games where humans are better? Probably so. But people don't watch these other games in prime time.
- moskie 16y ago
  Well, right, that wouldn't be Jeopardy, but ultimately, I don't really care about Jeopardy, thus my question. What I (and most of us here, I think) would find interesting is how Watson's language processing abilities compares to that of (very smart) people. I don't really care how fast Watson can ring a buzzer. If there was a way to gauge that, I'd find it interesting.
  - kenjackson 16y ago
    I assume you mean "natural" language processing. For that trivia is the wrong test. It's better to just give it paragraphs and ask it questions about the paragraph. And of course have increasing levels of difficult. More like the SAT reading comprehension test. This would extract how it stores data brought in from structured/unstructed sources and the actual task of processing natural language. I suspect it will be a fair bit worse at processing natural language than even the average person.

warmfuzzykitten 16y ago

Died with his buzzer in his hand, Lord Lord... Nice image by Jennings. But I was surprised to hear him complain about the buzzer reaction time advantage. In an earlier interview with the Washington Post he discounted this, saying that the best human players anticipate the timing while Watson has to wait for the signal. "On any given night, nearly all the contestants know nearly all the answers, so it's just a matter of who masters buzzer rhythm the best... But I wouldn't call this unfair...precise timing just happens to be one thing computers are better at than we humans. It's not like I think Watson should try buzzing in more erratically just to give homo sapiens a chance." http://live.washingtonpost.com/jeopardy-ken-jennings.html?hpid=talkbox1 http://live.washingtonpost.com/jeopardy-ken-jennings.html?hp...

sausagefeet 16y ago
Like sciolistse said I think the "unfair" part was a bit of editorializing.

protomyth 16y ago

I'd be more impressed by this massive IBM advertisement, if the computer received its information the exact way the other contestants did (verbally / visually looking at screen).

ddlatham 16y ago
It wouldn't make much difference. OCR on the text on the screen would be fine. If it had to do speech recognition for Alex Trebek, that would make it more difficult.
- protomyth 16y ago
  Yep, it would be more difficult and probably make its buzzer reaction time a little more "honest".
  - bryanlarsen 16y ago
    More honest? It would make it impossible, since if the computer had to parse Trebek's voice, it would finish getting the entire clue at the same time that the humans are pressing the buzzer.
  - kooshball 16y ago
    the OCR would be nearly perfect since there's very little variation in the way the text is displayed. The voice part wouldn't matter since the all top level players use the visual clue to determine the answer long before Trebek finishes reading it.
jessedhillon 16y ago
Are you the same guy who left this comment: http://www.crunchgear.com/2011/02/14/angry-nerds-ibms-watson-finishes-night-one-of-jeopardy-challenge-tied-for-first-place/#comment-146803262 http://www.crunchgear.com/2011/02/14/angry-nerds-ibms-watson...
- protomyth 16y ago

sosuke 16y ago

I think I remember reading that Watson didn't buzz in until it was sure it had an answer which was giving the human opponents an advantage because they could buzz in based of thinking that by the time they are asked for their answer they can have come up with the correct one. Looks like they must have overcome that issue.

deleted 16y ago

[deleted]

hyko 16y ago

John Henry beat the machine he was up against: http://en.wikipedia.org/wiki/John_Henry_(folklore) http://en.wikipedia.org/wiki/John_Henry_(folklore)

CWuestefeld 16y ago

From a competition perspective, I think it was fair: every player is entitled to capitalize his own advantages within the rules. However, I think this would have been more interesting intellectually to isolate Watson's "thinking", with everything else on a level playing field.

ajscherer 16y ago

His analogy about Kasparov needing to solve a long division problem is whack. Buzzing is part of the game of Jeapordy. For all we know he wouldn't have been a champion in a version of the game that didn't involve buzzing.

ConceptDog 16y ago

Ugh. He should have skipped writing about this. No matter the validity of his words, whatever he says will be percieved as sour grapes.

juddlyon 16y ago

No use in complaining, this was a cool PR stunt. You don't hear the Washington Generals bitching about the Globetrotters.