4 ms·

> The natural language processing is its biggest weakness. There were several documentaries about Watson after it won Jeopardy, and this weakness was quite ev

> The natural language processing is its biggest weakness. There were several documentaries about Watson after it won Jeopardy, and this weakness was quite evident to those paying attention. Watson never really understood Alex's Jeopardy "answers" at all. This was quite obvious in "final Jeopardy". Watson's response clearly showed its limitations. Here's how it went down:[1] The category was US Cities, and the answer was: “Its largest airport was named for a World War II hero; its second largest, for a World War II battle.” The two human contestants wrote “What is Chicago?” for its O’Hare and Midway, but Watson’s response was a lame “What is Toronto???” Basically Watson's Jeopardy responses were a very refined equivalent of the Google "I'm feeling lucky" button. I have to believe that the situation gets a lot lot trickier in a "medical context". Still, couldn't it be possible to harness Watson as a super-smart assistant to humans? But there's probably not enough money to be made in that. [1] http://asmarterplanet.com/blog/2011/02/watson-on-jeopardy-day-two-the-confusion-over-an-airport-clue.html http://asmarterplanet.com/blog/2011/02/watson-on-jeopardy-da...

TimPC 12y ago
In all fairness the Toronto answer isn't as ridiculous as it seems at first blush. If you confuse Lester B. Pearson with Alastair Pearson (something quite possible by NLP when you have to match "Pearson international airport" then you at least match you end up with a British World War II hero. Not being a U.S. city would involve either underfitting or ignoring category information, and trying to stretch buttonville or billy bishop into a WWII battle takes work, but it seems like precisely the kind of error that would be caused by actual NLP rather than optimizing page rank variants. I also think IBM has the right model by creating Watson as a service on 30% revenue share -> Let external developers find all the various products and business models. They'll be better at finding and optimizing new products than a services companies would be and some of them will be able to go after much smaller products. You end up with more diversification and less of IBM's capital at risk for 30% of what's likely to be a much larger pie than IBM can generate on their own.
- desdiv 12y ago
  The confusion probably arose from the Billy Bishop Toronto City Airport, named after the World War ONE flying ace, Billy Bishop.
- PhantomGremlin 12y ago
  It would be very interesting to find out how Watson actually interpreted both the category and the "answer". I don't know the actual amounts of time contestants are given, but on TV the final category is disclosed many minutes ahead of the "answer". Then a contestant has perhaps 30 seconds to formulate a "question". An eternity for a massively parallel computer like Watson. Quite different from the rest of the show, which relies on lightning reflexes when played by "champions". Viewed from 30,000 feet (quite appropriate for an airport question, eh) I think that Watson didn't understand the category. Watson did not know what the words "US cities" meant. Those aren't words that would normally have any sort of double meaning, so if Watson understood the category, why would it have "ignored" category information? That doesn't make sense, especially for final Jeopardy. As Dr. Venkman might say: "good guess, but wrong!" In Watson's defense, it did not have very high confidence in its answer. We'll probably never know the real story. That's not the kind of information that IBM would want to disclose, mainly because it would probably make Watson look bad.
- x0x0 12y ago
  I always get dubious when people suggest letting external developers find business models / use cases. I think if a product were truly compelling, they'd be able to find (and polish) at least one really awesome use case internally.
graycat 12y ago
When I was a researcher at IBM's Watson lab in Yorktown Heights, a standard explanation for many of the research projects was to give "luster" to the company. It's easy to suspect that the main goal of the Watson computer playing Jeopardy was really just luster. Really, there were some severely negative attitudes on (1) Research working on projects that could result in revenue and (2) transferring projects from Research to the rest of the company for development or sales. Our project in Research was quite exceptional and transferred two projects, both of which went to customers and got sold, but our group was unusual, and our success was not wanted by the higher ups. At one time Gerstner mentioned "all the exciting projects coming out of Research" -- right, about like the wings about to sprout on all those pigs. The explanation I like about IBM is one given to me by the manager of the Chicago branch office: "You might think of IBM as an electronics company or a computer company, but you would be wrong. IBM is a marketing company, and it would get into the grocery business tomorrow if it saw a business opportunity there. You might think that Research comes up with new product designs, development turns them into products, and marketing sells them, but that is exactly backwards. Instead, marketing figures out what they can sell, development builds it, and they go to Research if necessary." So, maybe the current changes amount to IBM trying some new things to market.

TimPC 12y ago

In all fairness the Toronto answer isn't as ridiculous as it seems at first blush. If you confuse Lester B. Pearson with Alastair Pearson (something quite possible by NLP when you have to match "Pearson international airport" then you at least match you end up with a British World War II hero. Not being a U.S. city would involve either underfitting or ignoring category information, and trying to stretch buttonville or billy bishop into a WWII battle takes work, but it seems like precisely the kind of error that would be caused by actual NLP rather than optimizing page rank variants. I also think IBM has the right model by creating Watson as a service on 30% revenue share -> Let external developers find all the various products and business models. They'll be better at finding and optimizing new products than a services companies would be and some of them will be able to go after much smaller products. You end up with more diversification and less of IBM's capital at risk for 30% of what's likely to be a much larger pie than IBM can generate on their own.

desdiv 12y ago
The confusion probably arose from the Billy Bishop Toronto City Airport, named after the World War ONE flying ace, Billy Bishop.
PhantomGremlin 12y ago
It would be very interesting to find out how Watson actually interpreted both the category and the "answer". I don't know the actual amounts of time contestants are given, but on TV the final category is disclosed many minutes ahead of the "answer". Then a contestant has perhaps 30 seconds to formulate a "question". An eternity for a massively parallel computer like Watson. Quite different from the rest of the show, which relies on lightning reflexes when played by "champions". Viewed from 30,000 feet (quite appropriate for an airport question, eh) I think that Watson didn't understand the category. Watson did not know what the words "US cities" meant. Those aren't words that would normally have any sort of double meaning, so if Watson understood the category, why would it have "ignored" category information? That doesn't make sense, especially for final Jeopardy. As Dr. Venkman might say: "good guess, but wrong!" In Watson's defense, it did not have very high confidence in its answer. We'll probably never know the real story. That's not the kind of information that IBM would want to disclose, mainly because it would probably make Watson look bad.
x0x0 12y ago
I always get dubious when people suggest letting external developers find business models / use cases. I think if a product were truly compelling, they'd be able to find (and polish) at least one really awesome use case internally.

graycat 12y ago

When I was a researcher at IBM's Watson lab in Yorktown Heights, a standard explanation for many of the research projects was to give "luster" to the company. It's easy to suspect that the main goal of the Watson computer playing Jeopardy was really just luster. Really, there were some severely negative attitudes on (1) Research working on projects that could result in revenue and (2) transferring projects from Research to the rest of the company for development or sales. Our project in Research was quite exceptional and transferred two projects, both of which went to customers and got sold, but our group was unusual, and our success was not wanted by the higher ups. At one time Gerstner mentioned "all the exciting projects coming out of Research" -- right, about like the wings about to sprout on all those pigs. The explanation I like about IBM is one given to me by the manager of the Chicago branch office: "You might think of IBM as an electronics company or a computer company, but you would be wrong. IBM is a marketing company, and it would get into the grocery business tomorrow if it saw a business opportunity there. You might think that Research comes up with new product designs, development turns them into products, and marketing sells them, but that is exactly backwards. Instead, marketing figures out what they can sell, development builds it, and they go to Research if necessary." So, maybe the current changes amount to IBM trying some new things to market.