4 ms·

I'm a SWE at Watson Health. What do you think was so absurd about it? It definitely didn't explain the constraints and limitations well enough (it won't diagnos

I'm a SWE at Watson Health. What do you think was so absurd about it? It definitely didn't explain the constraints and limitations well enough (it won't diagnose any problem you can have), but for the cancer scenario the segment centered on, the Watson solution is indeed able to greatly assist doctors with a task that does not scale well for them.

ben_jones 9y ago
Unfortunately its an emotional argument for me as it led to a 30 minute discussion over the phone with my father on the limitations of AI and the current state of marketing from most technology products. Theirs at least one post in this thread that goes into detail on the limitations of Watson Health (it mentioned oversold capabilities that led to a medical institution wasting ~$60 million dollars, but you would be right to point out that I have no direct examples from memory on why the 60 minutes coverage itself was 'absurd'. If anything its the effectiveness of Watson marketing (and other companies marketing of AI) that gets to me and how it is even becoming pervasive in cultural icons like 60 minutes.
UnquietTinkerer 9y ago
I think the 60 Minutes segment was just marketing copy handed to them by IBM (how else could they talk for 20 minutes about cutting-edge ML without mentioning Google or Amazon at all?). Watson is introduced as "an AI" - singular - designed to play Jeopardy and then, after he wins, nobly re-purposed to medicine. Charlie Rose even says at one point that "they taught Watson to diagnose cancer", implying that it was the same Watson who won Jeopardy. The truth is the original Watson is on display in a glass case somewhere and the new "Watson" shares only the name and some of the underlying ML techniques. It might be easy to understand what Charlie really meant when you are already familiar with the project, but the non-technical viewer will walk away thinking of Watson as a single, coherent entity sitting in an IBM vault somewhere. Conflating ML-based expert systems with anthropomorphic AI is farcical, supporting the accusations that IBM is overselling its product. It is also irresponsible, because it fuels the idea in the popular mind that machine learning is creating independent intelligences rather than mere tools.
- pgodzin 9y ago
  Google and Amazon were presumably left out because the segment was in the context of the healthcare field. I don't think DeepMind's healthcare initiatives were very mature at that point. I agree the language is problematic for non-technical viewers, but at the same time these are difficult concepts to explain to non-technical people, and explain the details of what it can't do is obviously much less compelling than focusing on what it can do.
  - semi-extrinsic 9y ago
    > these are difficult concepts to explain to non-technical people But that's not the problem, is it? It's not that IBM is trying to explain something difficult. Rather, if IBM was trying to be honest, the box that won Jeopardy would be called Watson, and the box they're now trying to sell everyone and their dog would be called "Whitney, Watson's big sister" or whatever. At the very least, "Watson the third". By calling it the exact same name, you are trying to convince us it's the exact same thing just with more training. Which it's not. Others mentioned poisoning the well; round here the expression is "pissing in the well", which I think fits much better.
    - Latty 9y ago
      I mean, plenty of software gets rewritten and sold under the same name as an old version - I'm not really sure I see the difference there.
  - ben_jones 9y ago
    IIRC the segment was followed by a self-driving car piece at Carnegie Mellon. If thats true then it was a clearly a general A.I. / M.L. piece. I've seen this episode twice and its also possible that a month ago it was re-aired alongside other content.
comstock 9y ago
I think it would be useful to link to some scientific evaluation of Watson Health. I'd want to see: 0. A baseline measurement. 1. Comparison to the nearest compatible. 2. Comparison against a simple algorithm/expert system.
- pgodzin 9y ago
  That's exactly what we do with all our pilot clients before we go into production. I'm not sure that IBM publishes these results after the pilot is done though.
  - comstock 9y ago
    Then it's understandable that people would be suspicious of a puff piece on 60minutes which provides no substantive information.
    - pgodzin 9y ago
      The people who care about the exact numbers and the audience for the 60 minutes piece are very different groups
      - comstock 9y ago
        There's obviously overlap, because people here have shown a dislike to the 60 minutes show. Overall that probably helps the brand with some groups, and harms it with others (scientists, engineers) so you may have more work to convince them, though the puff pieces might help sell it at the exec level.
LoSboccacc 9y ago
After how much supervised learning driven by highly paid experts? Watson would be more credible if papers/people citing it's efficacy would start showing the actual queries and answers of the system without being constantly mediated by IBM oracles sitting between renaponses and their interpretations
- pgodzin 9y ago
  Not sure what you're asking. The hospitals usually need to provided a lot of doctor-annotated data for training. But the custom solution is whatever the clients want it to be, not necessarily a question-answering system like Jeopardy that you seem to be asking about.

ben_jones 9y ago

Unfortunately its an emotional argument for me as it led to a 30 minute discussion over the phone with my father on the limitations of AI and the current state of marketing from most technology products. Theirs at least one post in this thread that goes into detail on the limitations of Watson Health (it mentioned oversold capabilities that led to a medical institution wasting ~$60 million dollars, but you would be right to point out that I have no direct examples from memory on why the 60 minutes coverage itself was 'absurd'. If anything its the effectiveness of Watson marketing (and other companies marketing of AI) that gets to me and how it is even becoming pervasive in cultural icons like 60 minutes.

UnquietTinkerer 9y ago

I think the 60 Minutes segment was just marketing copy handed to them by IBM (how else could they talk for 20 minutes about cutting-edge ML without mentioning Google or Amazon at all?). Watson is introduced as "an AI" - singular - designed to play Jeopardy and then, after he wins, nobly re-purposed to medicine. Charlie Rose even says at one point that "they taught Watson to diagnose cancer", implying that it was the same Watson who won Jeopardy. The truth is the original Watson is on display in a glass case somewhere and the new "Watson" shares only the name and some of the underlying ML techniques. It might be easy to understand what Charlie really meant when you are already familiar with the project, but the non-technical viewer will walk away thinking of Watson as a single, coherent entity sitting in an IBM vault somewhere. Conflating ML-based expert systems with anthropomorphic AI is farcical, supporting the accusations that IBM is overselling its product. It is also irresponsible, because it fuels the idea in the popular mind that machine learning is creating independent intelligences rather than mere tools.

pgodzin 9y ago
Google and Amazon were presumably left out because the segment was in the context of the healthcare field. I don't think DeepMind's healthcare initiatives were very mature at that point. I agree the language is problematic for non-technical viewers, but at the same time these are difficult concepts to explain to non-technical people, and explain the details of what it can't do is obviously much less compelling than focusing on what it can do.
- semi-extrinsic 9y ago
  > these are difficult concepts to explain to non-technical people But that's not the problem, is it? It's not that IBM is trying to explain something difficult. Rather, if IBM was trying to be honest, the box that won Jeopardy would be called Watson, and the box they're now trying to sell everyone and their dog would be called "Whitney, Watson's big sister" or whatever. At the very least, "Watson the third". By calling it the exact same name, you are trying to convince us it's the exact same thing just with more training. Which it's not. Others mentioned poisoning the well; round here the expression is "pissing in the well", which I think fits much better.
  - Latty 9y ago
    I mean, plenty of software gets rewritten and sold under the same name as an old version - I'm not really sure I see the difference there.
- ben_jones 9y ago
  IIRC the segment was followed by a self-driving car piece at Carnegie Mellon. If thats true then it was a clearly a general A.I. / M.L. piece. I've seen this episode twice and its also possible that a month ago it was re-aired alongside other content.

comstock 9y ago

I think it would be useful to link to some scientific evaluation of Watson Health. I'd want to see: 0. A baseline measurement. 1. Comparison to the nearest compatible. 2. Comparison against a simple algorithm/expert system.

pgodzin 9y ago
That's exactly what we do with all our pilot clients before we go into production. I'm not sure that IBM publishes these results after the pilot is done though.
- comstock 9y ago
  Then it's understandable that people would be suspicious of a puff piece on 60minutes which provides no substantive information.
  - pgodzin 9y ago
    The people who care about the exact numbers and the audience for the 60 minutes piece are very different groups
    - comstock 9y ago
      There's obviously overlap, because people here have shown a dislike to the 60 minutes show. Overall that probably helps the brand with some groups, and harms it with others (scientists, engineers) so you may have more work to convince them, though the puff pieces might help sell it at the exec level.

LoSboccacc 9y ago

After how much supervised learning driven by highly paid experts? Watson would be more credible if papers/people citing it's efficacy would start showing the actual queries and answers of the system without being constantly mediated by IBM oracles sitting between renaponses and their interpretations

pgodzin 9y ago
Not sure what you're asking. The hospitals usually need to provided a lot of doctor-annotated data for training. But the custom solution is whatever the clients want it to be, not necessarily a question-answering system like Jeopardy that you seem to be asking about.