8 ms·

Why Ken Jennings’s ‘Jeopardy’ Streak Is Nearly Impossible to Break

100 points by mblevin 11y ago

ilaksh 11y ago
Unless you're a computer.
- deleted 11y ago
  [deleted]
jMyles 11y ago
It never occurred to me that optimized buzzer timing was such a big part of Watson's dominance. Kinda takes some of the romance out of it.
- ramidarigaz 11y ago
  Yeah, it definitely wasn't as impressive as it seemed on the surface. A much better test of Watson's performance would have been QuizBowl, where contestants can buzz in while the question is still being read.
  - fsk 11y ago
    They also restricted the clues compared to what they would normally have - no audio/video Daily Doubles. I think there might have been some other restrictions, but I don't recall the exact rules. I think they didn't allow any of the "special" categories they sometimes have. They also did some other tricks, like giving Watson a parsed ASCII stream of the clue, rather than making it see the screen and parse it out. I think it also got an electronic feed of the "you can buzz in now signal" rather than making it parse out the same light the contestants see.
    - guelo 11y ago
      So, to be fair, they should have added a delay to the "you can buzz in now signal" to match the slowness of the human visual processing system. Without that, if all players knew the answer Watson won every time. Also, the ASCII feed should have matched human reading speed.
    - c17r 11y ago
      Some detailed info about the timing and the buzzers: http://www.kurzweilai.net/the-buzzer-factor-did-watson-have-an-unfair-advantage http://www.kurzweilai.net/the-buzzer-factor-did-watson-have-...
- gamegoblin 11y ago
  Something I never liked about jeopardy is that you can't buzz in at any time. I played on a trivia team at the high school level, and most questions were structured like: "This American architect, born in 1867, is known for works such as the Robie House, the Guggenheim Museum, and Fallingwater." So they typically feed you information from least to most well known, and you can buzz in at any time. This rewards players who know the most trivia. Everyone who played knows Frank Lloyd Wright did Fallingwater, but a very good player would know his approximate date of birth and would buzz in immediately on hearing 1867. I understand that this might make Jeopardy less fun for the audience, but I feel it would be a better test of trivia instead of buzzer timing.
  - shasta 11y ago
    With Jeopardy questions, you'd buzz in at "This American architect..."
    - aaronbrethorst 11y ago
      That's what I did in quiz bowl in high school. Calculated risk, but one that's likely to pay off.
  - fsk 11y ago
    They made it that way on purpose, so it's more entertaining for the audience. If you could buzz in early, the players would almost always buzz in before Alex Trebek finished reading the question, which makes it less interesting for the audience. With the wait-to-buzz-in rule, the audience gets an chance to understand and maybe think of the answer before the contestants speak.
    - jsmthrowaway 11y ago
      I'd contend that the "signal when permitted" minigame makes Jeopardy! more difficult, too, because the part of your brain that arrives at the answer must yield to the part of your brain that tenses for a stimulus. I definitely have to switch "modes" between pulling up deeply buried knowledge and rapid reaction "stance", and I noticed this when I played a well-made copy of the show's rules complete with the signaling system. There's another side, too, which is if folks could signal early, reading fast would become an even stronger optimization. It already is, but at least everybody gets an equal chance the way it is (even if they got the answer fairly quickly from reading the clue while ignoring Trebek).
- femto 11y ago
  > And when Jennings went up against IBM’s Watson in a special competition, the computer’s biggest advantage was not its knowledge base but rather an optimized buzzer timing So the computer's most effective weapon against Jennings was a 555 timer? That's cute.
  - eru 11y ago
    Only because the rest was already optimized.
- scott_s 11y ago
  I think there's more to it than that. I actually played Watson on the Jeopardy set they filmed the Watson episodes on. That "set" was just the auditorium here at IBM Research in NY, and they left the set intact for about six months after taping. Colleagues of mine hosted an academic conference in our building during that time. Since Watson was on everyone's mind at that point (and we were literally looking at the set for it during people's talks), my colleagues asked the head systems researcher if he could give a special talk at the conference. He graciously accepted, and offered to do a full demonstration. There was only one volunteer from the rest of the conference - I think that if you're not a native English speaker, and not familiar with American pop culture and history, Jeopardy questions can be mystifying - so I gladly volunteered to play. To answer the obvious question: I lost. Badly. In fact, not only did I lose, I lost a full-cheating version of Jeopardy where the audience (participants from the conference) shouted out answers, and I buzzed in using answers I liked. Not only that, but some of the answers that the audience was shouting were taken from Watson, looking at the readout it showed of candidate answers. (Some of the people were able to immediately recognize the correct answer, even though Watson was unsure of it. But keep in mind Watson came up with those answers.) I would have lost to even a decent Jeopardy player, so losing to Watson is kind of overkill. But I think I learned a few things about playing Jeopardy, and how the human experience differs from Watson's experience. I spent almost all of my cycles when the host was asking the question trying to figure out when he would stop talking. That is, I spent the entire question-asking period timing my buzz. I could not think of the answer and time my buzz simultaneously. The few times I thought I did know the answer, I did not know the answer when I buzzed in. I knew I knew the answer. The difference is that I did not have the answer in mind when buzzing, just that given a few seconds - which you get, by the basic protocols of the game - I could come up with the answer. I think other human players are the same: they spend the question-asking period timing their buzz, and are going by an intuitive feel for whether or not they know the answer. The actual answer recall does not happen until after they buzz in. Watson is inverted. It only buzzes in when it is confident of its answer. In that sense, the game is not just about Watson winning the buzzer timing. Watson definitely could win every buzzer, but then it would lose the game; it's not accurate enough to get every question correct. So if Watson won every buzzer, it would answer incorrectly many times, and ultimately lose. With that in mind, I hope it restores some of the romance. During the test run, the category was "Presidential Rhymes" with the clue "Barack's pack animals". Watson answered "Obama's llama's" immediately. Me and the other contestant just stared at each other, slack-jawed. I had seen it done on tv, but there was something far more immediate and impressive, since the head systems designer was ten feet from me, and the Power7 systems just a few rooms away.
martin1975 11y ago
Ken Jennings' 74 win streak is because...? A. He cheated B. He's lucky C. He's a genius D. It's written
- jplarson 11y ago
  Seems harsh that so many people downvoted this, I dig the Slumdog Millionaire reference!
  - axitanull 11y ago
    I believe that is because many people don't want comment section in Hacker News turn into the one in Reddit, which consist of mostly jokes, pop culture references...things that doesn't really contribute to the discussion.
    - jMyles 11y ago
      > things that doesn't really contribute to the discussion. There may be a diversity in palette for this matter, even among the same people at different times.
    - jplarson 11y ago
      Got it, thanks!
- goback2reddit 11y ago
  Go back to reddit.
monochromatic 11y ago
> So we (or, more precisely, FiveThirtyEight’s lead lifestyle writer Walt Hickey) ran a simulation that flipped a weighted coin with a 97.9 percent chance of landing on a Jennings win. Every time it did, we “flipped the coin” again. We did 1 million of these simulations. This is just a binomial distribution. Did they seriously estimate the answer instead of solving it analytically?
- Hobotron1 11y ago
  I'm not certain about 538s target demographic these days. I'm gonna guess that they did solve it analytically, but for the sake of exposition and readers that might not know what a binomial distribution, they bring up the coin-flip as an illustrative example.
- allenlavoie 11y ago
  The waiting time distribution is actually the geometric distribution (with expectation 1/p = 1/0.021 = 47.6, so about what they got).
- jdoerrie 11y ago
  Given that we stop after the first failure technically this should be a negative binomial distribution (http://en.wikipedia.org/wiki/Negative_binomial_distribution http://en.wikipedia.org/wiki/Negative_binomial_distribution) with p = 0.979 and r = 1. The mean is p / (1-p) = 46.62. Also there is a p ^ 75 = 0.204 chance that someone of same skill will achieve a better score.
- tel 11y ago
  There's the didactic argument where they did it analytically but explained the intuition of the analysis mechanically. There's also the oversimplification argument where they actually did some kind of much larger sampling from a trained decision tree but then boiled the mechanism down to something easily understood by casual readers more interested in "Jeopardy!" than statistics.
datashovel 11y ago
I'm convinced that not only do we stand on the shoulders of giants, but we live among giants. What Ken Jennings accomplished is remarkable by any interpretation. How he did so probably even he can't explain.
Steko 11y ago
> Here’s how we figured this out. We assumed that Jennings headed into Final Jeopardy with at most one other player in contention to win. Then we assumed that in order for Jennings to lose, this sequence of events has to happen: The game must not be locked up, Jennings must get the Final Jeopardy question wrong, and the other contender must get it right. We used his stats across all 75 of his games, not just his 74 wins, to better reflect his overall skills. The probability he doesn’t have the game locked up is 13.3 percent, the probability he gets Final Jeopardy wrong is 32 percent, and the probability the other contender gets it right is 49 percent. When we combine these probabilities, we see that Jennings only has about a 2.1 percent chance of losing. This is incorrect and overstates his dominance because (1) it assumes Jennings will be ahead, something that won't always happen against Rutter/Jennings/Collins/Chu/Craig level players and (2) the lock up and opponent knowing Final Jeopardy conditions are not independent. Pretty sloppy basic probability by 538 and make me think I'm not missing much by not subscribing (they didn't offer full RSS last I checked).
- Aushin 11y ago
  When you say that they're not independent, do you mean that the probability that the person knows the Final Jeopardy question is higher because they've managed to keep Ken Jennings from running away with it (i.e. they're better than the average person Jennings has faced)? Just checking my understanding.

100 points by mblevin 11y ago

ilaksh 11y ago

Unless you're a computer.

deleted 11y ago
[deleted]

jMyles 11y ago

It never occurred to me that optimized buzzer timing was such a big part of Watson's dominance. Kinda takes some of the romance out of it.

ramidarigaz 11y ago
Yeah, it definitely wasn't as impressive as it seemed on the surface. A much better test of Watson's performance would have been QuizBowl, where contestants can buzz in while the question is still being read.
- fsk 11y ago
  They also restricted the clues compared to what they would normally have - no audio/video Daily Doubles. I think there might have been some other restrictions, but I don't recall the exact rules. I think they didn't allow any of the "special" categories they sometimes have. They also did some other tricks, like giving Watson a parsed ASCII stream of the clue, rather than making it see the screen and parse it out. I think it also got an electronic feed of the "you can buzz in now signal" rather than making it parse out the same light the contestants see.
  - guelo 11y ago
    So, to be fair, they should have added a delay to the "you can buzz in now signal" to match the slowness of the human visual processing system. Without that, if all players knew the answer Watson won every time. Also, the ASCII feed should have matched human reading speed.
  - c17r 11y ago
    Some detailed info about the timing and the buzzers: http://www.kurzweilai.net/the-buzzer-factor-did-watson-have-an-unfair-advantage http://www.kurzweilai.net/the-buzzer-factor-did-watson-have-...
gamegoblin 11y ago
Something I never liked about jeopardy is that you can't buzz in at any time. I played on a trivia team at the high school level, and most questions were structured like: "This American architect, born in 1867, is known for works such as the Robie House, the Guggenheim Museum, and Fallingwater." So they typically feed you information from least to most well known, and you can buzz in at any time. This rewards players who know the most trivia. Everyone who played knows Frank Lloyd Wright did Fallingwater, but a very good player would know his approximate date of birth and would buzz in immediately on hearing 1867. I understand that this might make Jeopardy less fun for the audience, but I feel it would be a better test of trivia instead of buzzer timing.
- shasta 11y ago
  With Jeopardy questions, you'd buzz in at "This American architect..."
  - aaronbrethorst 11y ago
    That's what I did in quiz bowl in high school. Calculated risk, but one that's likely to pay off.
- fsk 11y ago
  They made it that way on purpose, so it's more entertaining for the audience. If you could buzz in early, the players would almost always buzz in before Alex Trebek finished reading the question, which makes it less interesting for the audience. With the wait-to-buzz-in rule, the audience gets an chance to understand and maybe think of the answer before the contestants speak.
  - jsmthrowaway 11y ago
    I'd contend that the "signal when permitted" minigame makes Jeopardy! more difficult, too, because the part of your brain that arrives at the answer must yield to the part of your brain that tenses for a stimulus. I definitely have to switch "modes" between pulling up deeply buried knowledge and rapid reaction "stance", and I noticed this when I played a well-made copy of the show's rules complete with the signaling system. There's another side, too, which is if folks could signal early, reading fast would become an even stronger optimization. It already is, but at least everybody gets an equal chance the way it is (even if they got the answer fairly quickly from reading the clue while ignoring Trebek).
femto 11y ago
> And when Jennings went up against IBM’s Watson in a special competition, the computer’s biggest advantage was not its knowledge base but rather an optimized buzzer timing So the computer's most effective weapon against Jennings was a 555 timer? That's cute.
- eru 11y ago
  Only because the rest was already optimized.
scott_s 11y ago
I think there's more to it than that. I actually played Watson on the Jeopardy set they filmed the Watson episodes on. That "set" was just the auditorium here at IBM Research in NY, and they left the set intact for about six months after taping. Colleagues of mine hosted an academic conference in our building during that time. Since Watson was on everyone's mind at that point (and we were literally looking at the set for it during people's talks), my colleagues asked the head systems researcher if he could give a special talk at the conference. He graciously accepted, and offered to do a full demonstration. There was only one volunteer from the rest of the conference - I think that if you're not a native English speaker, and not familiar with American pop culture and history, Jeopardy questions can be mystifying - so I gladly volunteered to play. To answer the obvious question: I lost. Badly. In fact, not only did I lose, I lost a full-cheating version of Jeopardy where the audience (participants from the conference) shouted out answers, and I buzzed in using answers I liked. Not only that, but some of the answers that the audience was shouting were taken from Watson, looking at the readout it showed of candidate answers. (Some of the people were able to immediately recognize the correct answer, even though Watson was unsure of it. But keep in mind Watson came up with those answers.) I would have lost to even a decent Jeopardy player, so losing to Watson is kind of overkill. But I think I learned a few things about playing Jeopardy, and how the human experience differs from Watson's experience. I spent almost all of my cycles when the host was asking the question trying to figure out when he would stop talking. That is, I spent the entire question-asking period timing my buzz. I could not think of the answer and time my buzz simultaneously. The few times I thought I did know the answer, I did not know the answer when I buzzed in. I knew I knew the answer. The difference is that I did not have the answer in mind when buzzing, just that given a few seconds - which you get, by the basic protocols of the game - I could come up with the answer. I think other human players are the same: they spend the question-asking period timing their buzz, and are going by an intuitive feel for whether or not they know the answer. The actual answer recall does not happen until after they buzz in. Watson is inverted. It only buzzes in when it is confident of its answer. In that sense, the game is not just about Watson winning the buzzer timing. Watson definitely could win every buzzer, but then it would lose the game; it's not accurate enough to get every question correct. So if Watson won every buzzer, it would answer incorrectly many times, and ultimately lose. With that in mind, I hope it restores some of the romance. During the test run, the category was "Presidential Rhymes" with the clue "Barack's pack animals". Watson answered "Obama's llama's" immediately. Me and the other contestant just stared at each other, slack-jawed. I had seen it done on tv, but there was something far more immediate and impressive, since the head systems designer was ten feet from me, and the Power7 systems just a few rooms away.

martin1975 11y ago

Ken Jennings' 74 win streak is because...? A. He cheated B. He's lucky C. He's a genius D. It's written

jplarson 11y ago
Seems harsh that so many people downvoted this, I dig the Slumdog Millionaire reference!
- axitanull 11y ago
  I believe that is because many people don't want comment section in Hacker News turn into the one in Reddit, which consist of mostly jokes, pop culture references...things that doesn't really contribute to the discussion.
  - jMyles 11y ago
    > things that doesn't really contribute to the discussion. There may be a diversity in palette for this matter, even among the same people at different times.
  - jplarson 11y ago
    Got it, thanks!
goback2reddit 11y ago
Go back to reddit.

monochromatic 11y ago

> So we (or, more precisely, FiveThirtyEight’s lead lifestyle writer Walt Hickey) ran a simulation that flipped a weighted coin with a 97.9 percent chance of landing on a Jennings win. Every time it did, we “flipped the coin” again. We did 1 million of these simulations. This is just a binomial distribution. Did they seriously estimate the answer instead of solving it analytically?

Hobotron1 11y ago
I'm not certain about 538s target demographic these days. I'm gonna guess that they did solve it analytically, but for the sake of exposition and readers that might not know what a binomial distribution, they bring up the coin-flip as an illustrative example.
allenlavoie 11y ago
The waiting time distribution is actually the geometric distribution (with expectation 1/p = 1/0.021 = 47.6, so about what they got).
jdoerrie 11y ago
Given that we stop after the first failure technically this should be a negative binomial distribution (http://en.wikipedia.org/wiki/Negative_binomial_distribution http://en.wikipedia.org/wiki/Negative_binomial_distribution) with p = 0.979 and r = 1. The mean is p / (1-p) = 46.62. Also there is a p ^ 75 = 0.204 chance that someone of same skill will achieve a better score.
tel 11y ago
There's the didactic argument where they did it analytically but explained the intuition of the analysis mechanically. There's also the oversimplification argument where they actually did some kind of much larger sampling from a trained decision tree but then boiled the mechanism down to something easily understood by casual readers more interested in "Jeopardy!" than statistics.

datashovel 11y ago

I'm convinced that not only do we stand on the shoulders of giants, but we live among giants. What Ken Jennings accomplished is remarkable by any interpretation. How he did so probably even he can't explain.

Steko 11y ago

> Here’s how we figured this out. We assumed that Jennings headed into Final Jeopardy with at most one other player in contention to win. Then we assumed that in order for Jennings to lose, this sequence of events has to happen: The game must not be locked up, Jennings must get the Final Jeopardy question wrong, and the other contender must get it right. We used his stats across all 75 of his games, not just his 74 wins, to better reflect his overall skills. The probability he doesn’t have the game locked up is 13.3 percent, the probability he gets Final Jeopardy wrong is 32 percent, and the probability the other contender gets it right is 49 percent. When we combine these probabilities, we see that Jennings only has about a 2.1 percent chance of losing. This is incorrect and overstates his dominance because (1) it assumes Jennings will be ahead, something that won't always happen against Rutter/Jennings/Collins/Chu/Craig level players and (2) the lock up and opponent knowing Final Jeopardy conditions are not independent. Pretty sloppy basic probability by 538 and make me think I'm not missing much by not subscribing (they didn't offer full RSS last I checked).

Aushin 11y ago
When you say that they're not independent, do you mean that the probability that the person knows the Final Jeopardy question is higher because they've managed to keep Ken Jennings from running away with it (i.e. they're better than the average person Jennings has faced)? Just checking my understanding.