7 ms·

Interesting match. Seems like Watson was able to ring in (clicker) much quicker than Ken or Brad. Any unfair advantage?

marketer 16y ago
It is an unfair advantage, and it's frustrating to watch. For most of the answers, both Ken and Brad were trying to buzz in, but Watson always had better timing and buzzed in first. I'm sure that Jeopardy's buzzing system didn't take robots into account when it was designed, so it technically isn't against the rules. But it does give Watson a huge tactical advantage.
- harryh 16y ago
  I've been trying to think of a way to redesign the buzzer system to make it more fair, but I haven't been able to come up with anything. Tricky problem.
  - DannoHung 16y ago
    You could just give Watson a hard 200ms handicap.
    - gloob 16y ago
      So the game turns from "the absurdly smart computer stomps everyone" to "the absurdly smart (but deliberately crippled) computer may or may not stomp everyone, depending mostly on random chance." That doesn't sound like a great improvement, imho.
      - derefr 16y ago
        What about just removing the timing element altogether, effectively making every round into (possibly iterated) Final Jeopardy?
        gloob 16y ago
        That would give a more precise idea of just how many of the questions Watson was able to answer correctly, but at that point the discussion starts to feel like people proposing changes to the rules of chess in order to screw Deep Blue.
        baddox 16y ago
        That would be more "fair," by which I mean that it would remove the obviously computer-favored dynamic of buzzer reflexes. Of course, it wouldn't make good TV, it wouldn't really be Jeopardy!, and I suspect Watson would lose. Against normal season contestants Watson would probably fair well (in a buzzerless competition), but against human champions Watson wouldn't stand a chance.
        ugh 16y ago
        Do you know that? How? Stats?
        baddox 16y ago
        There's no way to know what portion of the clues the human contestants knew the correct responses to. I've watched Ken and Brad play extensively though, so I very strongly suspect that they easily knew at least as many as Watson.
        ugh 16y ago
        I would like some stats about that. As I said, Watson could have correctly answered 84% of the questions, nobody could answer 6% of the questions. That leaves a very thin margin for the humans, if past games are any indication one that is probably not large enough.
  - cperciva 16y ago
    How to make the buzzers fair: If two people buzz in within time T of the question being read, randomly select who gets to answer. The value of T would be somewhere on the order of 200 ms, to distinguish between the two cases of "waiting until the question is finished before buzzing in" and "racing to figure out the answer the fastest".
    - baddox 16y ago
      It would probably be better to just linearly go through who gets to answer.
      - ugh 16y ago
        That wouldn’t be Jeopardy anymore. Timing matters. I like the 200ms solution.
  - baddox 16y ago
    They could revert to the old rule where you could buzz in as soon as you thought you knew the answer. It's hard to tell how long it takes Watson to come up with answers right now, but switching to the old rule would introduce an interesting dynamic where Watson would have to decide whether to spend more time chugging data or buzz in. Also, this would be pretty artificial (no pun intended), but they could analyze previous all-human Jeopardy! episodes and figure out average buzzer response time, and perhaps incorporate that into Watson.
  - jim-greer 16y ago
    You could just change the game completely, and allow multiple people to answer simultaneously if they buzz in within a couple of seconds. It looked to me like Watson would be pretty competitive in that game, too.
  - gojomo 16y ago
    I wonder when Watson gets the textual clue: the instant the clue appears on screen? The moment Alex finishes reading it? A midpoint? Is it trickled word-by-word at about Alex's speaking or average human sight-reading rates? If Watson has confidence at the moment ringing-in is allowed, it seems it will always win. So how much time it has to achieve that confidence, as a function of how the text is fed, may be more important than buzzer mechanics. Based on the same general idea of a common starting point that motivates waiting until after the entire question has been read before allowing any ringing in, I could see there being a tiny 'common period' where all buzzes are considered as coming in simultaneously, with the person chosen to answer then being chosen in round-robin fashion (or for maximal drama, favoring whoever is behind). After the tiny common period, it would be strictly based on first-to-buzz. It would take away a twitch-timing factor that has been important for human champions, too, but offer more fairness with regard to computers and even people with slight ticks or timing problems.
    - Cixelyn 16y ago
      The clue is given to Watson electronically the moment the clue appears on the screen. While watching, I was actually hoping for really short questions which would cut down on Watson's time to process and possibly put Watson on more even footing with Jennings and Rutter.
ellyagg 16y ago
It is certainly unfair and has pretty much ruined the competition for me. Watson can instantly detect when he's allowed to ring in. This is no different than a car "racing" a human. In theory, a human contestant can try to time his trigger to coincide with the end of Trebeck speaking the question, and catch the light just as the timing person releases the question, but in practice this is fraught with peril--if you're off you can't ring in for several tenths of a second--and requires much more of a human's energy, which he can no longer devote to figuring out the question. You could see Jennings attempt this on the first day, with mixed results. These episodes are commercials for IBM and it's clear who is supposed to win.
- gloob 16y ago
  Not certain what's unfair about it - if we accept that the human contestants would have scored comparably to Watson (if they had the opportunity), then the competition really does boil down to who has the fastest thumb. I can't imagine it would be any more entertaining to watch IBM throw the match by intentionally crippling their player.
  - baddox 16y ago
    It's not unfair exactly, but it certainly ruins the impressiveness of Watson's ability to come up with the right answers. I highly doubt Watson's accuracy is even close to Brad or Ken's, but the buzzer seals the deal. It's less impressive, just like a computer sorting 1000 integers faster than a human is less impressive. edit: added "not" after the first word
    - gloob 16y ago
      From the descriptions I've read, Watson only buzzed in after it believed it had a solid answer. If it was significantly less accurate than its competitors, it wouldn't have spent the entire match buzzing in before them. I'm afraid I don't really understand the decision to trivialize the fact that we now have a computer that can answer general-knowledge natural-language queries quickly and about as accurately as a clever person. That's a Big Deal.
zach 16y ago
Of course. This isn't a match of equals, this is one of man versus machine. As fans know, most responses in a game of Jeopardy! are known by multiple players. That's especially so in a game of this caliber, so the knowledge aspect is really quite minimal compared to the ring-in factor. Both these guys slaughtered their opponents by being quick on the buzzer. Watson is being granted first crack at the questions 90% of the time because of its electromechanical advantage. IBM may not have the mean brainpower that Google has, but they can clearly build a computer that can press a button quicker than Ken Jennings. Knowing that a computer can consistently beat even the best to ever play the game to the buzzer, the IBM team could be pretty well assured of success once they got Watson performing well enough.
- sigstoat 16y ago
  the comparison to google seems, um, awkward? "IBM holds more patents than any other U.S.-based technology company and has nine research laboratories worldwide. Its employees have garnered five Nobel Prizes, four Turing Awards, nine National Medals of Technology, and five National Medals of Science." admittedly, they've been around longer, but they're not exactly playing with crayons over there.
  - zach 16y ago
    I'm just cracking wise. Watson is a very impressive project. Nonetheless, throw four to seven Google researchers and engineers at this project and you'd achieve comparable results. They've been cornering the market in this area. But instant button pressing is not terribly impressive and yet is Watson's critical advantage. This match is framed as a battle of knowledge, not a struggle for humans to overcome the massive disadvantage of their meat-based nervous system, which it is. This is largely glossed over. Watson would lose horribly if one of its engineers was pressing the button. If its electronic advantage on the button was taken away, it would be the kind of fascinating match I and a lot of people were expecting.
    - _ques 16y ago
      Google actually does do open-domain fact-based question answering for queries like: "what is the capital of china" which are probably closest to Jeopardy questions (technically speaking.) The team that does this is a lot more than seven Google engineers. Yet, it is not the top selling features of Google, because -- it's not that easy. There is a fine line between "seems easy" and "nearly impossible". A Google engineer does not equate to infinite skills.
baddox 16y ago
For anyone who understands the true dynamics of Jeopardy!, Watson simply isn't impressive as a Jeopardy! contestant as it's being made out to be. I'm not minimizing the awesomeness of Watson's language processing, because it's great, but Watson's performance on Jeopardy! is only as impressive as a hypothetical competition where Watson and two humans fill out a worksheet and compare who got the most answers right. During tournaments of champions, Jeopardy! is not about how many correct responses you can come up with, as all competitors will know the vast majority of them. It's all about timing. I don't know the exact statistics, but it seemed like Watson knew about 75% of the correct responses. I strongly suspect the two human contestants knew a greater percentage. On day 2, it just came down to timing. Watson was only beaten to the buzzer three times when it knew the correct response.
- ugh 16y ago
  Watson correctly knew 84% of the answers (26 of 31). It actually answered 74% (23 of 31). It didn’t know the correct response or was not confident enough to answer 16% of the time (5 of 31). Nobody at all could give an answer to 6% of all questions (2 of 31). If the two questions we know nobody could answer are excluded, Watson knew the right answer 90% of the time (26 of 29). Do individual Jeopardy contestants correctly know more than 84% of the answers? Two questions were left unanswered correctly by everyone, 93% (29 of 31) is already the upper bound for the human performance in this round, that’s not very far from 84%. Looking at the six rounds of Jeopardy the J! Archive has of games with Ken Jennings and Brad Rutter [0] we see that the average upper bound in those six rounds was 87% or 26 questions (of 30). The minimum was 23 and the maximum 28 (of 30). That’s, again, very much an upper bound. Looking at all this, I’m pretty confident that Watson would be doing well even if it had human reaction times. (This is how I would change the game: Within 200ms or so – whatever the human reaction time on a Jeopardy buzzer is — after the buzzers are open, the player to answer is randomly selected from all who managed to buzz in. After those 200ms everything stays the same. Players are also not punished for buzzing in too early. This is a minimal change to the game, not a completely different game which I think is important.) [0] http://www.j-archive.com/showplayer.php?player_id=7206 http://www.j-archive.com/showplayer.php?player_id=7206
  - 16y ago
InclinedPlane 16y ago
You're glossing over the fact that Watson knew the correct answer most of the time. That's what this is about, and that's why this is a significant result.
- baddox 16y ago
  It looked like Watson had the correct response about 75% of the time. That's probably better than most normal season contestants and home viewers, but I am highly confident that Ken and Brad knew significantly more than 75% of the correct responses.
  - ugh 16y ago
    84% of the time.
  - InclinedPlane 16y ago
    Doesn't matter, you're discussing a different set of goals. Which might make an interesting game, but ignores the purpose of this exercise. The question on everyone's mind was "can a machine beat humans at Jeopardy". The answer appears to be: yes, very much so. Whether or not we can make a more interesting or "fair" game of Jeopardy involving such a machine is an entirely separate, and to my mind far less interesting, question. Where do we go from here? It would seem silly to say "we make a more interesting Humans vs. machines Jeopardy game." Rather, it seems more prudent to figure out ways to expand on this research and use the underlying technology to solve more interesting, and practical, problems.

It is an unfair advantage, and it's frustrating to watch. For most of the answers, both Ken and Brad were trying to buzz in, but Watson always had better timing and buzzed in first. I'm sure that Jeopardy's buzzing system didn't take robots into account when it was designed, so it technically isn't against the rules. But it does give Watson a huge tactical advantage.

harryh 16y ago
I've been trying to think of a way to redesign the buzzer system to make it more fair, but I haven't been able to come up with anything. Tricky problem.
- DannoHung 16y ago
  You could just give Watson a hard 200ms handicap.
  - gloob 16y ago
    So the game turns from "the absurdly smart computer stomps everyone" to "the absurdly smart (but deliberately crippled) computer may or may not stomp everyone, depending mostly on random chance." That doesn't sound like a great improvement, imho.
    - derefr 16y ago
      What about just removing the timing element altogether, effectively making every round into (possibly iterated) Final Jeopardy?
      - gloob 16y ago
        That would give a more precise idea of just how many of the questions Watson was able to answer correctly, but at that point the discussion starts to feel like people proposing changes to the rules of chess in order to screw Deep Blue.
      - baddox 16y ago
        That would be more "fair," by which I mean that it would remove the obviously computer-favored dynamic of buzzer reflexes. Of course, it wouldn't make good TV, it wouldn't really be Jeopardy!, and I suspect Watson would lose. Against normal season contestants Watson would probably fair well (in a buzzerless competition), but against human champions Watson wouldn't stand a chance.
        ugh 16y ago
        Do you know that? How? Stats?
        baddox 16y ago
        There's no way to know what portion of the clues the human contestants knew the correct responses to. I've watched Ken and Brad play extensively though, so I very strongly suspect that they easily knew at least as many as Watson.
        ugh 16y ago
        I would like some stats about that. As I said, Watson could have correctly answered 84% of the questions, nobody could answer 6% of the questions. That leaves a very thin margin for the humans, if past games are any indication one that is probably not large enough.
- cperciva 16y ago
  How to make the buzzers fair: If two people buzz in within time T of the question being read, randomly select who gets to answer. The value of T would be somewhere on the order of 200 ms, to distinguish between the two cases of "waiting until the question is finished before buzzing in" and "racing to figure out the answer the fastest".
  - baddox 16y ago
    It would probably be better to just linearly go through who gets to answer.
    - ugh 16y ago
      That wouldn’t be Jeopardy anymore. Timing matters. I like the 200ms solution.
- baddox 16y ago
  They could revert to the old rule where you could buzz in as soon as you thought you knew the answer. It's hard to tell how long it takes Watson to come up with answers right now, but switching to the old rule would introduce an interesting dynamic where Watson would have to decide whether to spend more time chugging data or buzz in. Also, this would be pretty artificial (no pun intended), but they could analyze previous all-human Jeopardy! episodes and figure out average buzzer response time, and perhaps incorporate that into Watson.
- jim-greer 16y ago
  You could just change the game completely, and allow multiple people to answer simultaneously if they buzz in within a couple of seconds. It looked to me like Watson would be pretty competitive in that game, too.
- gojomo 16y ago
  I wonder when Watson gets the textual clue: the instant the clue appears on screen? The moment Alex finishes reading it? A midpoint? Is it trickled word-by-word at about Alex's speaking or average human sight-reading rates? If Watson has confidence at the moment ringing-in is allowed, it seems it will always win. So how much time it has to achieve that confidence, as a function of how the text is fed, may be more important than buzzer mechanics. Based on the same general idea of a common starting point that motivates waiting until after the entire question has been read before allowing any ringing in, I could see there being a tiny 'common period' where all buzzes are considered as coming in simultaneously, with the person chosen to answer then being chosen in round-robin fashion (or for maximal drama, favoring whoever is behind). After the tiny common period, it would be strictly based on first-to-buzz. It would take away a twitch-timing factor that has been important for human champions, too, but offer more fairness with regard to computers and even people with slight ticks or timing problems.
  - Cixelyn 16y ago
    The clue is given to Watson electronically the moment the clue appears on the screen. While watching, I was actually hoping for really short questions which would cut down on Watson's time to process and possibly put Watson on more even footing with Jennings and Rutter.

ellyagg 16y ago

It is certainly unfair and has pretty much ruined the competition for me. Watson can instantly detect when he's allowed to ring in. This is no different than a car "racing" a human. In theory, a human contestant can try to time his trigger to coincide with the end of Trebeck speaking the question, and catch the light just as the timing person releases the question, but in practice this is fraught with peril--if you're off you can't ring in for several tenths of a second--and requires much more of a human's energy, which he can no longer devote to figuring out the question. You could see Jennings attempt this on the first day, with mixed results. These episodes are commercials for IBM and it's clear who is supposed to win.

gloob 16y ago
Not certain what's unfair about it - if we accept that the human contestants would have scored comparably to Watson (if they had the opportunity), then the competition really does boil down to who has the fastest thumb. I can't imagine it would be any more entertaining to watch IBM throw the match by intentionally crippling their player.
- baddox 16y ago
  It's not unfair exactly, but it certainly ruins the impressiveness of Watson's ability to come up with the right answers. I highly doubt Watson's accuracy is even close to Brad or Ken's, but the buzzer seals the deal. It's less impressive, just like a computer sorting 1000 integers faster than a human is less impressive. edit: added "not" after the first word
  - gloob 16y ago
    From the descriptions I've read, Watson only buzzed in after it believed it had a solid answer. If it was significantly less accurate than its competitors, it wouldn't have spent the entire match buzzing in before them. I'm afraid I don't really understand the decision to trivialize the fact that we now have a computer that can answer general-knowledge natural-language queries quickly and about as accurately as a clever person. That's a Big Deal.

zach 16y ago

Of course. This isn't a match of equals, this is one of man versus machine. As fans know, most responses in a game of Jeopardy! are known by multiple players. That's especially so in a game of this caliber, so the knowledge aspect is really quite minimal compared to the ring-in factor. Both these guys slaughtered their opponents by being quick on the buzzer. Watson is being granted first crack at the questions 90% of the time because of its electromechanical advantage. IBM may not have the mean brainpower that Google has, but they can clearly build a computer that can press a button quicker than Ken Jennings. Knowing that a computer can consistently beat even the best to ever play the game to the buzzer, the IBM team could be pretty well assured of success once they got Watson performing well enough.

sigstoat 16y ago
the comparison to google seems, um, awkward? "IBM holds more patents than any other U.S.-based technology company and has nine research laboratories worldwide. Its employees have garnered five Nobel Prizes, four Turing Awards, nine National Medals of Technology, and five National Medals of Science." admittedly, they've been around longer, but they're not exactly playing with crayons over there.
- zach 16y ago
  I'm just cracking wise. Watson is a very impressive project. Nonetheless, throw four to seven Google researchers and engineers at this project and you'd achieve comparable results. They've been cornering the market in this area. But instant button pressing is not terribly impressive and yet is Watson's critical advantage. This match is framed as a battle of knowledge, not a struggle for humans to overcome the massive disadvantage of their meat-based nervous system, which it is. This is largely glossed over. Watson would lose horribly if one of its engineers was pressing the button. If its electronic advantage on the button was taken away, it would be the kind of fascinating match I and a lot of people were expecting.
  - _ques 16y ago
    Google actually does do open-domain fact-based question answering for queries like: "what is the capital of china" which are probably closest to Jeopardy questions (technically speaking.) The team that does this is a lot more than seven Google engineers. Yet, it is not the top selling features of Google, because -- it's not that easy. There is a fine line between "seems easy" and "nearly impossible". A Google engineer does not equate to infinite skills.

baddox 16y ago

For anyone who understands the true dynamics of Jeopardy!, Watson simply isn't impressive as a Jeopardy! contestant as it's being made out to be. I'm not minimizing the awesomeness of Watson's language processing, because it's great, but Watson's performance on Jeopardy! is only as impressive as a hypothetical competition where Watson and two humans fill out a worksheet and compare who got the most answers right. During tournaments of champions, Jeopardy! is not about how many correct responses you can come up with, as all competitors will know the vast majority of them. It's all about timing. I don't know the exact statistics, but it seemed like Watson knew about 75% of the correct responses. I strongly suspect the two human contestants knew a greater percentage. On day 2, it just came down to timing. Watson was only beaten to the buzzer three times when it knew the correct response.

ugh 16y ago
Watson correctly knew 84% of the answers (26 of 31). It actually answered 74% (23 of 31). It didn’t know the correct response or was not confident enough to answer 16% of the time (5 of 31). Nobody at all could give an answer to 6% of all questions (2 of 31). If the two questions we know nobody could answer are excluded, Watson knew the right answer 90% of the time (26 of 29). Do individual Jeopardy contestants correctly know more than 84% of the answers? Two questions were left unanswered correctly by everyone, 93% (29 of 31) is already the upper bound for the human performance in this round, that’s not very far from 84%. Looking at the six rounds of Jeopardy the J! Archive has of games with Ken Jennings and Brad Rutter [0] we see that the average upper bound in those six rounds was 87% or 26 questions (of 30). The minimum was 23 and the maximum 28 (of 30). That’s, again, very much an upper bound. Looking at all this, I’m pretty confident that Watson would be doing well even if it had human reaction times. (This is how I would change the game: Within 200ms or so – whatever the human reaction time on a Jeopardy buzzer is — after the buzzers are open, the player to answer is randomly selected from all who managed to buzz in. After those 200ms everything stays the same. Players are also not punished for buzzing in too early. This is a minimal change to the game, not a completely different game which I think is important.) [0] http://www.j-archive.com/showplayer.php?player_id=7206 http://www.j-archive.com/showplayer.php?player_id=7206
- 16y ago

InclinedPlane 16y ago

You're glossing over the fact that Watson knew the correct answer most of the time. That's what this is about, and that's why this is a significant result.

baddox 16y ago
It looked like Watson had the correct response about 75% of the time. That's probably better than most normal season contestants and home viewers, but I am highly confident that Ken and Brad knew significantly more than 75% of the correct responses.
- ugh 16y ago
  84% of the time.
- InclinedPlane 16y ago
  Doesn't matter, you're discussing a different set of goals. Which might make an interesting game, but ignores the purpose of this exercise. The question on everyone's mind was "can a machine beat humans at Jeopardy". The answer appears to be: yes, very much so. Whether or not we can make a more interesting or "fair" game of Jeopardy involving such a machine is an entirely separate, and to my mind far less interesting, question. Where do we go from here? It would seem silly to say "we make a more interesting Humans vs. machines Jeopardy game." Rather, it seems more prudent to figure out ways to expand on this research and use the underlying technology to solve more interesting, and practical, problems.