6 ms·

I'm quite late to this thread, but I worked on Watson very briefly (not on the core development, but overall system performance improvements). I think there's

I'm quite late to this thread, but I worked on Watson very briefly (not on the core development, but overall system performance improvements). I think there's a major misunderstanding of Watson which isn't helped by IBM's Marketing efforts. IBM Marketing has been slapping the "Cognitive" label on everything and is creating unrealistic expectations. The Jeopardy playing Watson (DeepQA pipeline) was a landmark success at Information Retrieval, its architecture is built largely on Apache UIMA and Lucene with proprietary code for scaling out (performance) and filtering & ranking. I'm not an expert on IR so I won't comment further. This is very different from Neural Nets that are all the rage in ML today. I'd like to point the following links from David Ferrucci [1] the original architect of Watson and this technical publication at aaai.org [2]. The DeepQA pipeline wasn't fluff, the intention was to take this question-answer pipeline and apply it to other verticals such as Law and Medicine, essentially replace the Jeopardy playing Watson's corpus of Wikipedia, Britannica etc... with Legal and Medical equivalents. Given its runaway PR success, the Watson brand was applied to many other areas which haven't been successful but I'd like to point out what the original product was here. [1] https://www.singularityweblog.com/david-ferrucci-on-singularity-1-on-1-pursue-the-big-challenges/ https://www.singularityweblog.com/david-ferrucci-on-singular... [2] https://www.aaai.org/Magazine/Watson/watson.php https://www.aaai.org/Magazine/Watson/watson.php

nl 9y ago
This is true. I've read all the Watson Special Edition papers numerous times[1]. The original Watson work was very good, and adding neural-network based models as alternative question parsers and answer rankers is a clear way forward. It's a pity IBM decided to use the Watson name for everthing they do in ML now. [1] http://brenocon.com/watson_special_issue/ http://brenocon.com/watson_special_issue/
denzil_correa 9y ago
> The Jeopardy playing Watson (DeepQA pipeline) was a landmark success at Information Retrieval, its architecture is built largely on Apache UIMA and Lucene with proprietary code for scaling out (performance) and filtering & ranking. I'm not an expert on IR so I won't comment further. This is very different from Neural Nets that are all the rage in ML today. The question I would like to ask is that if the architecture is built on UIMA and Lucence, what exactly did "Watson" do?
- filereaper 9y ago
  Apache UIMA came out of the efforts from Ferrucci and others from TJ Watson Research Lab. "Watson's" secret sauce was the filtering and ranking, you get multiple results from the models, which ones do you pick? There was significant engineering effort put into reducing the latency of getting answers out of the pipeline as well, you had to optimize given the constraints of latency, accuracy and confidence from the pipeline.
- dmreedy 9y ago
  If you can find them, read the papers on Question Analysis and Answer Coercion. I am still impressed by how powerful these systems were, and how effective. UIMA is a framework for tying algorithms together, and experimenting quickly with different combinations. (locked-down IEEE versions: http://ieeexplore.ieee.org/xpl/tocresult.jsp?isnumber=6177717 http://ieeexplore.ieee.org/xpl/tocresult.jsp?isnumber=617771...)
- nl 9y ago
  There's a lot more there than just search. See http://brenocon.com/watson_special_issue/ http://brenocon.com/watson_special_issue/
- ezy 9y ago
  "Watson" was a Jeopardy playing bot. The main trick behind Watson is to take the various systems (parsers, search, et al.) and hacks (constraints imposed by the rules of jeopardy) needed by a jeopardy playing bot and put them all together. So, in some sense, you could say UIMA is what Watson did -- because it allowed a lot of flexibility for researchers to combine their efforts. Ranking and filtering becomes of ultimate importance in a system like that because at some point you have to make a decision. However, it is terribly reliant on the other modules at least getting somewhere in the ballpark -- and the ranking is also not, by itself, anything impressive. So, it's an interesting case of how far you can get by just setting a single goal and slamming everything together -- but as it turns out, for every new domain you wish to apply something like that to, that magic ballpark is hard to reach without a significant amount of engineering & research effort to come up with new systems/hacks combined with a lot of relevant data. In other words, just like any other adhoc AIish system with a particular goal. Change the goal, change the system. So, of course Watson was oversold, it was a PR and Sales effort from the beginning. Sort of like AlphaGo or DeepBlue -- you might be able to find one or two interesting ideas in the bowels of such a system -- but the system itself is not a generic one.
avereveard 9y ago
> there's a major misunderstanding of Watson which isn't helped by IBM's Marketing efforts you got cause effect wrong, since it's a glorified search engine, the marketing team did all they could do pass this off as intelligence and to muddle the water around what Watson could and couldn't do. when sales target still were falling short, they started dumping everything under the Watson budget to prop up the dept.
- deleted 9y ago
  [deleted]
spydum 9y ago
I have seen this pattern repeated in many large companies. One big product is successful and builds a nice PR bubble. Executives pile on anything they had in the pipeline with a re-brand, diluting the new brand. I wonder what the solution is? Stronger brand owners within the company who can defend and be the quality control of what gets associated? Or is it hopeless?
- aweiland 9y ago
  MS did this with .NET. They even tried to brand a version of Windows with it for a while. I heard at one point no on in MS could really tell you want .NET was anymore.
cirgue 9y ago
IBM has definitely created some cool stuff at various times. The problem, though, is that IBM misrepresents themselves in publicly-facing marketing and with paying clients during implementation.

nl 9y ago

This is true. I've read all the Watson Special Edition papers numerous times[1]. The original Watson work was very good, and adding neural-network based models as alternative question parsers and answer rankers is a clear way forward. It's a pity IBM decided to use the Watson name for everthing they do in ML now. [1] http://brenocon.com/watson_special_issue/ http://brenocon.com/watson_special_issue/

denzil_correa 9y ago

> The Jeopardy playing Watson (DeepQA pipeline) was a landmark success at Information Retrieval, its architecture is built largely on Apache UIMA and Lucene with proprietary code for scaling out (performance) and filtering & ranking. I'm not an expert on IR so I won't comment further. This is very different from Neural Nets that are all the rage in ML today. The question I would like to ask is that if the architecture is built on UIMA and Lucence, what exactly did "Watson" do?

filereaper 9y ago
Apache UIMA came out of the efforts from Ferrucci and others from TJ Watson Research Lab. "Watson's" secret sauce was the filtering and ranking, you get multiple results from the models, which ones do you pick? There was significant engineering effort put into reducing the latency of getting answers out of the pipeline as well, you had to optimize given the constraints of latency, accuracy and confidence from the pipeline.
dmreedy 9y ago
If you can find them, read the papers on Question Analysis and Answer Coercion. I am still impressed by how powerful these systems were, and how effective. UIMA is a framework for tying algorithms together, and experimenting quickly with different combinations. (locked-down IEEE versions: http://ieeexplore.ieee.org/xpl/tocresult.jsp?isnumber=6177717 http://ieeexplore.ieee.org/xpl/tocresult.jsp?isnumber=617771...)
nl 9y ago
There's a lot more there than just search. See http://brenocon.com/watson_special_issue/ http://brenocon.com/watson_special_issue/
ezy 9y ago
"Watson" was a Jeopardy playing bot. The main trick behind Watson is to take the various systems (parsers, search, et al.) and hacks (constraints imposed by the rules of jeopardy) needed by a jeopardy playing bot and put them all together. So, in some sense, you could say UIMA is what Watson did -- because it allowed a lot of flexibility for researchers to combine their efforts. Ranking and filtering becomes of ultimate importance in a system like that because at some point you have to make a decision. However, it is terribly reliant on the other modules at least getting somewhere in the ballpark -- and the ranking is also not, by itself, anything impressive. So, it's an interesting case of how far you can get by just setting a single goal and slamming everything together -- but as it turns out, for every new domain you wish to apply something like that to, that magic ballpark is hard to reach without a significant amount of engineering & research effort to come up with new systems/hacks combined with a lot of relevant data. In other words, just like any other adhoc AIish system with a particular goal. Change the goal, change the system. So, of course Watson was oversold, it was a PR and Sales effort from the beginning. Sort of like AlphaGo or DeepBlue -- you might be able to find one or two interesting ideas in the bowels of such a system -- but the system itself is not a generic one.

avereveard 9y ago

> there's a major misunderstanding of Watson which isn't helped by IBM's Marketing efforts you got cause effect wrong, since it's a glorified search engine, the marketing team did all they could do pass this off as intelligence and to muddle the water around what Watson could and couldn't do. when sales target still were falling short, they started dumping everything under the Watson budget to prop up the dept.

deleted 9y ago
[deleted]

spydum 9y ago

I have seen this pattern repeated in many large companies. One big product is successful and builds a nice PR bubble. Executives pile on anything they had in the pipeline with a re-brand, diluting the new brand. I wonder what the solution is? Stronger brand owners within the company who can defend and be the quality control of what gets associated? Or is it hopeless?

aweiland 9y ago
MS did this with .NET. They even tried to brand a version of Windows with it for a while. I heard at one point no on in MS could really tell you want .NET was anymore.

cirgue 9y ago

IBM has definitely created some cool stuff at various times. The problem, though, is that IBM misrepresents themselves in publicly-facing marketing and with paying clients during implementation.