9 ms·
Watson crushes the competition in second round of 'Jeopardy'
- jamesjyu 16y agoPart 1 of the second round on YouTube here: http://www.youtube.com/watch?v=PHhDLUVAtqU http://www.youtube.com/watch?v=PHhDLUVAtqU
- rockstar9 16y agois there a replay of jeopardy?
- mashmac2 16y agoNormally, Jeopardy shows new shows on Weeknights. Popular shows are reshown on weekends. Also, around holidays and other non-normal weeks of broadcasting, older shows are re-aired. I don't believe episodes are available legally online.
- philsalesses 16y agoThey are out there. Be industrious, you'll find them.
- jackowayed 16y agoIBM will have it up in a couple days: http://twitter.com/#!/IBMWatson/status/37223337453158400 http://twitter.com/#!/IBMWatson/status/37223337453158400
- ssclafani 16y agoDavid Ferrucci, the manager of the Watson project at IBM, on why he thinks Watson got the Final Jeopardy question wrong: "First, the category names on Jeopardy! are tricky. The answers often do not exactly fit the category. Watson, in his training phase, learned that categories only weakly suggest the kind of answer that is expected, and, therefore, the machine downgrades their significance. The way the language was parsed provided an advantage for the humans and a disadvantage for Watson, as well. “What US city” wasn’t in the question. If it had been, Watson would have given US cities much more weight as it searched for the answer. Adding to the confusion for Watson, there are cities named Toronto in the United States and the Toronto in Canada has an American League baseball team. It probably picked up those facts from the written material it has digested. Also, the machine didn’t find much evidence to connect either city’s airport to World War II. (Chicago was a very close second on Watson’s list of possible answers.) So this is just one of those situations that’s a snap for a reasonably knowledgeable human but a true brain teaser for the machine." http://asmarterplanet.com/blog/2011/02/watson-on-jeopardy-day-two-the-confusion-over-an-airport-clue.html http://asmarterplanet.com/blog/2011/02/watson-on-jeopardy-da...
- gojomo 16y agoLame excuses! Watson is impressive, but I'm disappointed by the lack of any hint of comprehension behind its answers. When it's wrong, it's often nonsensically out-to-lunch, and its 2nd/3rd best answers are also often batty. If it's just trained-up on statistical correlations between trigger phrases and likely answers in the constrained Jeopardy domain, then 90 32-core/512GB RAM servers seem like overkill.
- InclinedPlane 16y agoWatson also generates confidence estimates and minimum confidence bars for questions. It may sometimes have batty "answers" but usually it knows they are batty. The rate of incorrect answers that Watson has a high confidence in is fairly low. What's remarkable and important to not take lightly is the result that it's possible to generate answers to often vague and indirect clues without understanding. That likely means that it will be possible to build useful systems for automating research and the synthesis of large amounts of data without needing to build artificial human-level intelligence.
- kirpekar 16y agoInteresting match. Seems like Watson was able to ring in (clicker) much quicker than Ken or Brad. Any unfair advantage?
- marketer 16y agoIt is an unfair advantage, and it's frustrating to watch. For most of the answers, both Ken and Brad were trying to buzz in, but Watson always had better timing and buzzed in first. I'm sure that Jeopardy's buzzing system didn't take robots into account when it was designed, so it technically isn't against the rules. But it does give Watson a huge tactical advantage.
- harryh 16y agoI've been trying to think of a way to redesign the buzzer system to make it more fair, but I haven't been able to come up with anything. Tricky problem.
- DannoHung 16y agoYou could just give Watson a hard 200ms handicap.
- gloob 16y agoSo the game turns from "the absurdly smart computer stomps everyone" to "the absurdly smart (but deliberately crippled) computer may or may not stomp everyone, depending mostly on random chance." That doesn't sound like a great improvement, imho.
- derefr 16y agoWhat about just removing the timing element altogether, effectively making every round into (possibly iterated) Final Jeopardy?
- 16y ago
- philsalesses 16y agoToday, I learned that there are 7 cities in America named Toronto.
- theycallmemorty 16y agoI wonder how many of them partially fit the mold of having an airport named after WWII battles/heroes?
- philsalesses 16y agoI have 31% confidence there is something there...
- spitfire 16y agoSo is there any information on how they actually implemented watson? My understanding is it's a bayesian machine learning system, but I still don't know how it parses answers, or really does its magic. Also, if there is anyone who thinks silicon valley has the smartest people around, this type of stuff should change your mind. Facebook is short trousers compared to this. and it's just a tech demo.
- teraflop 16y agoThere's a high-level overview in this paper from AI Magazine, Fall 2010: http://www.stanford.edu/class/cs124/AIMagzine-DeepQA.pdf http://www.stanford.edu/class/cs124/AIMagzine-DeepQA.pdf
- jdale27 16y agoAlso, if there is anyone who thinks silicon valley has the smartest people around, this type of stuff should change your mind. Watson is an impressive achievement, but there are quite a few companies in Silicon Valley whose engineers could pull this off. It's more a matter of how much money management feels like throwing at it. It's great publicity for IBM, which has to put in a lot more effort than most Silicon Valley companies in order to look cool, but can afford it.
- spitfire 16y agoEntirely true. But the point I was trying to make is that the current batch of dotcoms isn't even in the same league as the bigboys. IBM, HP, SGI(?), part of oracle, google, parts of oracle. There's driven and then there's Smart.
- spitfire 16y agoAlso, does anyone else really love that architecture at the Watson research center? That just looks like a place I'd want to work. Lots of wood, stone and glass. Love it. Wired had some details on the building and the cafeteria is right out of mad men. So much better than "open plan" (We're too cheap to give code monkeys space). Here are the pictures: http://www.wired.com/epicenter/2011/02/watson-jeopardy/?pid=204&viewall=true http://www.wired.com/epicenter/2011/02/watson-jeopardy/?pid=...
- baddox 16y agoAnd, as I predicted, it only came to buzzer reflex, which computers unsurprisingly excel at. On day 2 (today), Watson was only beaten to the buzzer three times when it had the correct response above its confidence threshold.
- e40 16y agoWhy the spoiler? Do you think everyone watches it live? We are in the age of the DVR.
- bpodgursky 16y ago<SPOILER ALERT> Mubarak's not president of Egypt anymore!
- deleted 16y ago[deleted]
- baddox 16y agoI don't think so. I would hope the clue writers were purposefully made unaware which clues were being written for the Watson episodes. I don't think they seemed unfair, either. They seemed like normal modern Jeopardy! clues. Also, I am highly confident that if you simply had Ken, Brad, and Watson fill out a worksheet with all the clues written on it (thus bypassing the central game dynamic of buzzer reflexes), Watson would get the lowest score.
- akashs 16y agoI seem to be the only one missing this, so just wondering how it's been concluded that it was strictly due to timing? Watson didn't miss many and the one's it didn't answer, it had the correct answer most of the time. Just wondering how that means it would have gotten the lowest score but managed to beat everyone to the clicker. Am I missing something?
- cryptoz 16y agoWatson has no clicker advantage. There is a human backstage that decides when Alex Trebek has finished speaking, and he lights up a board so the players know when they're allowed to click in. Watson gets the same signal and has to also press a physical button. There is a 300 ms delay penalty if you click in too early, and there was a question in tonight's show that seemed very much like Watson knew the answer but couldn't click in. He may have been estimating the clicker and getting it wrong. While his mechanical finger might be a few ms faster than a human hand, it does not provide any significant advantage to him.
- w01fe 16y agoHuman reaction time to visual stimuli is approximately 150 to 300 ms [1]. I don't know how Watson was configured, but even with the same visual cue I would guess it could press the button in a few milliseconds. That's a massive advantage, enough to win the buzz-in every time both a human and the machine know the answer by the time the light goes on. [1] http://en.wikipedia.org/wiki/Reflex http://en.wikipedia.org/wiki/Reflex
- emelski 16y agoFor all the talk of the difficulties of playing Jeopardy! due to the "nuances of natural language" and "puns and double meanings in the clues", that did not really seem to be a factor in the second round -- most of the questions were quite plainly worded with answers easily discoverable just by searching. Accordingly, Watson performed dramatically better today than yesterday, when a larger portion of the questions did have nuance and plays-on-words in the phrasing. Note too how spectacularly badly Watson performed on the Final Jeopardy! question, where nuance _did_ play a much bigger role. So today, we learned that machines can push buttons faster than people, and search is a great way to find answers for trivia questions. I doubt the former is a surprise to anybody alive in the past 50 years; the latter shouldn't surprise anybody who's ever used Google.
- gthank 16y agoThis. A thousand times this. Watson absolutely CRUSHED the human players on pretty much every question that was basic facts. I know Watson can probably generate answers faster than humans on simple search stuff, but it seemed so bad at some points that I wondered: is Watson not wired in with some sort of delay that mimics the delay that humans have between deciding to buzz in and actually buzzing in? A lack of such a system would seem to skew the results somewhat.
- effigies 16y agoI was at a watching party with a couple of IBMers who worked on Watson, and one thing they said is that it's not a question of speed, but of timing. Players time their pressing to an estimate of when Alex will finish the question, and Brad Rutter in particular has been clocked at under 2ms with shocking regularity. The advantages Watson has are consistency and the emotional perturbations in its opponents. You could see them getting frustrated, and that likely only served to harm their ability to hit that window between the end of the question and Watson's button press.
- gthank 16y agoYou're right: consistently being 6x faster on the buzzer than the common case for your opponent is going to let you destroy them. Their only hope is that you can't come up with a response before Alex finishes reading the question. I arrived at the 6x approximation by googling around for avg. ethernet latencies. I'm consistently seeing numbers of .3 - .35 ms for an ethernet ping/pong. I think it's fair to assume that with the money IBM has invested in this, Watson is on at least ethernet quality connections.
- to 16y agoanyone else thinks google could produce a better watson on a weekend? google because of the bigtable access to basically everything ever written (books, news, meds, tweets, groups, oh and the whole friggin internet)
- nyellin 16y agoI received a few complaints when I posted the results of round #1 and it hit the homepage. You might want to change the title to something ambiguous about who won.
- ckwalsh 16y agoJust put up the latest results https://spreadsheets1.google.com/ccc?key=tth_jhM8vyBAuogqHllHmHQ#gid=2 https://spreadsheets1.google.com/ccc?key=tth_jhM8vyBAuogqHll...
- drstrangevibes 16y agothe fact that watson has good nlp isnt nearly as impressive as the fact that it has a huge knowledge base, how the hell did it get all that knowledge, if it is just from browsing the internet by itself that makes me afraid.......very afraid
- metaprinter 16y agoI had to leave the article to search where the actual building was located because all they gave was, "suburban New York". I'm still not sure where it is.
- Curly 16y agoTo make it a true test of brains, and remove the mechanics of button-pressing speed from the question... Place all three contestants in isolation from each other. All three hear the question read, and buzz-in just as they do now. Allow ALL contestants who buzz in to answer the question, but do not allow them to know about their opponents' performances. Record all contestants' buzz-in reaction times. At the end of the game, compare only the accuracy of answers to determine the winner. At the end of the game, compare buzz-in reaction times to see how thumbs fare against relays.