6 ms·

Computer scientist Roger Craig built an app to prepare for Jeopardy domination.

157 points by soy714 15y ago

Most recent tournament champion, Roger Craig, broke Ken Jenning's all-time money record.<p>His finest display of Jeopardy prowess was this back-to-back "Daily Double" all-ins. http://www.youtube.com/watch?v=YRwK8SyVeJE<p>To prepare for the competition Craig developed a web app to analyze all the questions in Jeopardy's archive and reverse engineer the game. He explains it here. http://vimeo.com/29001512

klochner 15y ago
I'd like to see Roger go up against Watson, since both were "trained" for Jeopardy
- loumf 15y ago
  One advantage Roger has is insane betting (which could backfire, obviously). I didn't watch Watson -- did it bet so aggressively on Daily Double questions?
  - ssharp 15y ago
    Watson was able to analyze the current situation and make a rational bet on Daily Doubles. It analyzed it's comfort in the category the Daily Double was in, and then used a regression model to pick the optimal bet size, which factored in how winning or losing the Daily Double would change Watson's chances of winning or losing the game. I'm not really sure if Watson's model goes as far to consider other players' Daily Double behavior, or simply examines other players' scores. It would make sense to try and look at the game theory aspect though, especially in an iterative-game scenario. If a player bets hyper-aggressively on Daily Doubles (and Final Jeopardy) and gets most of them right, you'd think Watson would want to take that into consideration.
- nonsequ 15y ago
  My bet is that he'd lose. Beyond a certain level of knowledge, successful Jeopardy is all about hitting the buzzer first. Watson has a huge advantage in that respect because it gets the signal to buzz in electrically.
  - johno215 15y ago
    This. The Watson show was a not a completely fair match-up since Watson had a much better average buzzer time than his human opponents. If this was not the case, I would speculate a really good human opponent could beat Watson at least some of the time.
    - FaceKicker 15y ago
      It's hard to say if that really makes it "unfair", since a lot of Ken Jennings' (and every other Jeopardy champion's) success was due to his reaction time with the buzzer. That's simply a huge part of the game. On the other hand, nobody would dispute that a machine has faster reactions than a human, so maybe that makes it less interesting. Tough to figure out a great way to resolve that.
      - bradleyt 15y ago
        If you had data indicating what Ken Jennings' reaction time was, you could probably calibrate Watson to roughly match that.
        jerf 15y ago
        Yes, but is that fair? What is fair? Defining fair in terms of equal outcome is vacuous here. For a bit I thought it would be more interesting to let the humans ring in as soon as they had read the question, but the problem is that today they would slaughter Watson, and three years from now Watson would slaughter them, and possibly without even updating its software. I suppose the most fair thing to do would be to do away with the buzzer entirely. Let all players choose whether or not to answer all questions (that is, all three could get it right, all could get it wrong, one right, one abstain, one wrong, etc), and check the scores at the end. Without the buzzer advantage for Watson I suspect Jennings and Rutter still win... for another year or two, anyhow.
      - nonsequ 15y ago
        Having actually played against Watson, it certainly feels unfair when you've got a response you're 100% sure of on the tip of your tongue but you have to wait several long seconds for the question to be read. All the while you're desperately aware that Watson's reaction time is always 100 ms and yours ranges from 50-200 ms. It is interesting to watch machines encroach on things we once considered exclusively human mental abilities. In the early days processors just multiplied large numbers faster than we could. Then they beat a grandmaster at chess and now a machine is definitely better at Jeopardy.
zitterbewegung 15y ago
The only thing I can get from this article is that he created a study tool that wasn't a set of papers or a document but actually a computer program. I don't see how this has any significance other than he created a computer program. I wish there was more information to this article other than fluff.
- mpanzarino 15y ago
  The information is in the second video, I figured, why transcript Craig's description when he did a succinct job of it himself? To be honest, there's little information other than what he divulged out there on the net, as it was a private tool. If I find more i'll write a followup, it's pretty fascinating.
- soy714 15y ago
  There's a video of Roger Craig explaining the web app he created: http://vimeo.com/29001512 http://vimeo.com/29001512
  - DanielRibeiro 15y ago
    The wikipedia article contains a lot of detailed info on how well he did: http://en.wikipedia.org/wiki/Roger_Craig_%28Jeopardy!_contestant%29 http://en.wikipedia.org/wiki/Roger_Craig_%28Jeopardy!_contes...
- danteembermage 15y ago
  I think the big innovation highlighted in the video is that he used historical data on frequency of certain topics, the dollar value assigned to those topics, and his personal performance on certain categories to figure out which categories were most important to study. I know that I am bad at fashion and 19th century French literature but what relative fraction should I spend on one over the other? His code gave him a number and that's the ratio he studied them. So you're right, at that point it's not much different from flipping through a giant study manual, other than the manual telling you should should really look at pages 3, 11, 21, 47, 96, ... because that's what the solution to the nonlinear optimization problem said would give you the best chance at winning against simulated opponents.
zach 15y ago
Wow. This is a far more thorough approach than the experience-focused app I made in 2007, but then again Roger Craig is also a lot better at Jeopardy! than I am. http://zachbaker.com/how-to-win-on-jeopardy-with-ruby-on-rails http://zachbaker.com/how-to-win-on-jeopardy-with-ruby-on-rai...
georgieporgie 15y ago
I somehow missed the data-scraping introduction, that he sourced a fan-operated website that archives questions and answers. Here I'd thought he'd just off-handedly created a phenomenal data scraper for the general web. He didn't talk much about the algorithm he used to present questions for optimal memorization. I imagine it was spaced repetition, as used in SuperMemo: http://www.wired.com/medtech/health/magazine/16-05/ff_wozniak?currentPage=all http://www.wired.com/medtech/health/magazine/16-05/ff_woznia...
- gwern 15y ago
  Yes, that's right. That was my own question; he said he uses Anki: http://quantifiedself.com/2011/09/roger-craig-on-knowledge-tracking/#comment-3004 http://quantifiedself.com/2011/09/roger-craig-on-knowledge-t...
- mpobrien 15y ago
  A while back I actually built a mobile site for playing Jeopardy (originally conceived as a drinking game based on jeopardy, but that's another story...) part of which was a scraper that pulled data off the fan-sourced site and parsed it out. Code for parsing, if anyone's interested: https://github.com/mpobrien/jeoparty/blob/master/scrape_mongo2.py https://github.com/mpobrien/jeoparty/blob/master/scrape_mong... And code for scraping: https://github.com/mpobrien/jeoparty/blob/master/curler.sh https://github.com/mpobrien/jeoparty/blob/master/curler.sh
adaml_623 15y ago
"Of course, if he’s smart enough to build a focused learning app that allowed him to dominate the show like this, it’s probably likely he would have done ok on his own." The author of the article just doesn't get it.

157 points by soy714 15y ago

klochner 15y ago

I'd like to see Roger go up against Watson, since both were "trained" for Jeopardy

loumf 15y ago
One advantage Roger has is insane betting (which could backfire, obviously). I didn't watch Watson -- did it bet so aggressively on Daily Double questions?
- ssharp 15y ago
  Watson was able to analyze the current situation and make a rational bet on Daily Doubles. It analyzed it's comfort in the category the Daily Double was in, and then used a regression model to pick the optimal bet size, which factored in how winning or losing the Daily Double would change Watson's chances of winning or losing the game. I'm not really sure if Watson's model goes as far to consider other players' Daily Double behavior, or simply examines other players' scores. It would make sense to try and look at the game theory aspect though, especially in an iterative-game scenario. If a player bets hyper-aggressively on Daily Doubles (and Final Jeopardy) and gets most of them right, you'd think Watson would want to take that into consideration.
nonsequ 15y ago
My bet is that he'd lose. Beyond a certain level of knowledge, successful Jeopardy is all about hitting the buzzer first. Watson has a huge advantage in that respect because it gets the signal to buzz in electrically.
- johno215 15y ago
  This. The Watson show was a not a completely fair match-up since Watson had a much better average buzzer time than his human opponents. If this was not the case, I would speculate a really good human opponent could beat Watson at least some of the time.
  - FaceKicker 15y ago
    It's hard to say if that really makes it "unfair", since a lot of Ken Jennings' (and every other Jeopardy champion's) success was due to his reaction time with the buzzer. That's simply a huge part of the game. On the other hand, nobody would dispute that a machine has faster reactions than a human, so maybe that makes it less interesting. Tough to figure out a great way to resolve that.
    - bradleyt 15y ago
      If you had data indicating what Ken Jennings' reaction time was, you could probably calibrate Watson to roughly match that.
      - jerf 15y ago
        Yes, but is that fair? What is fair? Defining fair in terms of equal outcome is vacuous here. For a bit I thought it would be more interesting to let the humans ring in as soon as they had read the question, but the problem is that today they would slaughter Watson, and three years from now Watson would slaughter them, and possibly without even updating its software. I suppose the most fair thing to do would be to do away with the buzzer entirely. Let all players choose whether or not to answer all questions (that is, all three could get it right, all could get it wrong, one right, one abstain, one wrong, etc), and check the scores at the end. Without the buzzer advantage for Watson I suspect Jennings and Rutter still win... for another year or two, anyhow.
    - nonsequ 15y ago
      Having actually played against Watson, it certainly feels unfair when you've got a response you're 100% sure of on the tip of your tongue but you have to wait several long seconds for the question to be read. All the while you're desperately aware that Watson's reaction time is always 100 ms and yours ranges from 50-200 ms. It is interesting to watch machines encroach on things we once considered exclusively human mental abilities. In the early days processors just multiplied large numbers faster than we could. Then they beat a grandmaster at chess and now a machine is definitely better at Jeopardy.

zitterbewegung 15y ago

The only thing I can get from this article is that he created a study tool that wasn't a set of papers or a document but actually a computer program. I don't see how this has any significance other than he created a computer program. I wish there was more information to this article other than fluff.

mpanzarino 15y ago
The information is in the second video, I figured, why transcript Craig's description when he did a succinct job of it himself? To be honest, there's little information other than what he divulged out there on the net, as it was a private tool. If I find more i'll write a followup, it's pretty fascinating.
soy714 15y ago
There's a video of Roger Craig explaining the web app he created: http://vimeo.com/29001512 http://vimeo.com/29001512
- DanielRibeiro 15y ago
  The wikipedia article contains a lot of detailed info on how well he did: http://en.wikipedia.org/wiki/Roger_Craig_%28Jeopardy!_contestant%29 http://en.wikipedia.org/wiki/Roger_Craig_%28Jeopardy!_contes...
danteembermage 15y ago
I think the big innovation highlighted in the video is that he used historical data on frequency of certain topics, the dollar value assigned to those topics, and his personal performance on certain categories to figure out which categories were most important to study. I know that I am bad at fashion and 19th century French literature but what relative fraction should I spend on one over the other? His code gave him a number and that's the ratio he studied them. So you're right, at that point it's not much different from flipping through a giant study manual, other than the manual telling you should should really look at pages 3, 11, 21, 47, 96, ... because that's what the solution to the nonlinear optimization problem said would give you the best chance at winning against simulated opponents.

zach 15y ago

Wow. This is a far more thorough approach than the experience-focused app I made in 2007, but then again Roger Craig is also a lot better at Jeopardy! than I am. http://zachbaker.com/how-to-win-on-jeopardy-with-ruby-on-rails http://zachbaker.com/how-to-win-on-jeopardy-with-ruby-on-rai...

georgieporgie 15y ago

I somehow missed the data-scraping introduction, that he sourced a fan-operated website that archives questions and answers. Here I'd thought he'd just off-handedly created a phenomenal data scraper for the general web. He didn't talk much about the algorithm he used to present questions for optimal memorization. I imagine it was spaced repetition, as used in SuperMemo: http://www.wired.com/medtech/health/magazine/16-05/ff_wozniak?currentPage=all http://www.wired.com/medtech/health/magazine/16-05/ff_woznia...

gwern 15y ago
Yes, that's right. That was my own question; he said he uses Anki: http://quantifiedself.com/2011/09/roger-craig-on-knowledge-tracking/#comment-3004 http://quantifiedself.com/2011/09/roger-craig-on-knowledge-t...
mpobrien 15y ago
A while back I actually built a mobile site for playing Jeopardy (originally conceived as a drinking game based on jeopardy, but that's another story...) part of which was a scraper that pulled data off the fan-sourced site and parsed it out. Code for parsing, if anyone's interested: https://github.com/mpobrien/jeoparty/blob/master/scrape_mongo2.py https://github.com/mpobrien/jeoparty/blob/master/scrape_mong... And code for scraping: https://github.com/mpobrien/jeoparty/blob/master/curler.sh https://github.com/mpobrien/jeoparty/blob/master/curler.sh

adaml_623 15y ago

"Of course, if he’s smart enough to build a focused learning app that allowed him to dominate the show like this, it’s probably likely he would have done ok on his own." The author of the article just doesn't get it.