4 ms·

Preach on. I'm baffled by how techies can be so underwhelmed by what IBM is doing.

giardini 14y ago
Such software has been available for decades, been shown to be superior to human diagnosis, and yet has not been accepted by the medical community. Doctors will likely continue to push back. It may require restructuring of the medical education community (perhaps a' la the movie Idiocracy) to bring about utilization. "50 years of successful predictive modeling should be enough: Lessons for philosophy of science (2002)" http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.217.949 http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.217.... Idiocracy, the movie http://www.imdb.com/title/tt0387808/ http://www.imdb.com/title/tt0387808/
- tluyben2 14y ago
  I didn't read your links yet (on mobile), but I will. I know of great knowledge base systems from the 80s which were never accepted but performed very well. A UK PhD who now makes his money in pharma told me that, although the results were better, those systems were effectively blocked by the physician community (anyone has any citation/references?).
  - gwern 14y ago
    Legal issues and medical opposition was the explanation I read for why http://en.wikipedia.org/wiki/MYCIN http://en.wikipedia.org/wiki/MYCIN never went into practical use.
- MoosePlissken 14y ago
  The book "Thinking, Fast and Slow" has a great chapter on human predictions versus those made by forumlas. The superiority of simple forumlas was being demonstrated way back in the 50s (see "Clinical Versus Statistical Prediction: A Theoretical Analysis And A Review Of The Evidence"). Since then the evidence for statistical rather than intuitive prediction has only become stronger. From "Thinking, Fast and Slow": "The number of studies reporting comparisons of clinical and statistical predictions has increased to roughly two hundred, but the score in the contest between algorithms and humans has not changed. About 60% of the studies have shown significantly better accuracy for the algorithms. The other comparisons scored a draw in accuracy, but a tie is tantamount to a win for the statistical rules, which are normally much less expensive to use than expert judgment. No exception has been convincingly documented." Also interesting: "Several studies have shown that human decision makers are inferior to a prediction formula even when they are given the score suggested by the formula! They feel that they can overrule the formula because they have additional information about the case, but they are wrong more often than not."
  - bgilroy26 14y ago
    Just to pitch in 2c, I think society's reticence to implement automated decision making has much more to do with how to handle responsibility for mistakes. I think that is ultimately more important to people than picking the decoder with the best success rate. By distributing responsibility over many doctors (who hail from many medical schools), you make the pain more manageable. The concentrated responsibility that would result from having a single decision system make many many care decisions might be very difficult to stomach.
  - DeepDuh 14y ago
    60% out of 200 studies is not all that good, really. It allows the experts to believe that if they're still in the top 25%, no machine can beat them (I know that this is not exactly what the studies say, but statistics are almost always being misinterpreted, so I do it here too). The thing is: I expect Watson to perform significantly better than this. If it can beat Jeopardy grandmasters consistantly and if that technology is correctly being reapplied to the medical field, Watson should easily outperform anything we had before.
    - sesqu 14y ago
      It's worth noting that Watson didn't beat the Jeopardy grandmasters with accuracy, but with speed - not that that isn't useful in healthcare, given sufficient accuracy.
- sovande 14y ago
  True, so called expert systems are nothing new, but the much improved ability to parse natural language and semantics is newish. It means the human-machine interface is easier to use and combined with portable access via tablets this might very well be the tipping point. Similar to how the iPad was adopted compared to decades of tablets before that never saw a wide-spread adoption, mainly because of the poor human-machine interface.
rtkwe 14y ago
Not all of us. I'm graduating this spring and the top of my fucking awesome hire list was the Watson group in SWG @ IBM.
joonix 14y ago
Mining tweets to determine global reaction to a stadium power outage is far more interesting, of course.

Such software has been available for decades, been shown to be superior to human diagnosis, and yet has not been accepted by the medical community. Doctors will likely continue to push back. It may require restructuring of the medical education community (perhaps a' la the movie Idiocracy) to bring about utilization. "50 years of successful predictive modeling should be enough: Lessons for philosophy of science (2002)" http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.217.949 http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.217.... Idiocracy, the movie http://www.imdb.com/title/tt0387808/ http://www.imdb.com/title/tt0387808/

tluyben2 14y ago
I didn't read your links yet (on mobile), but I will. I know of great knowledge base systems from the 80s which were never accepted but performed very well. A UK PhD who now makes his money in pharma told me that, although the results were better, those systems were effectively blocked by the physician community (anyone has any citation/references?).
- gwern 14y ago
  Legal issues and medical opposition was the explanation I read for why http://en.wikipedia.org/wiki/MYCIN http://en.wikipedia.org/wiki/MYCIN never went into practical use.
MoosePlissken 14y ago
The book "Thinking, Fast and Slow" has a great chapter on human predictions versus those made by forumlas. The superiority of simple forumlas was being demonstrated way back in the 50s (see "Clinical Versus Statistical Prediction: A Theoretical Analysis And A Review Of The Evidence"). Since then the evidence for statistical rather than intuitive prediction has only become stronger. From "Thinking, Fast and Slow": "The number of studies reporting comparisons of clinical and statistical predictions has increased to roughly two hundred, but the score in the contest between algorithms and humans has not changed. About 60% of the studies have shown significantly better accuracy for the algorithms. The other comparisons scored a draw in accuracy, but a tie is tantamount to a win for the statistical rules, which are normally much less expensive to use than expert judgment. No exception has been convincingly documented." Also interesting: "Several studies have shown that human decision makers are inferior to a prediction formula even when they are given the score suggested by the formula! They feel that they can overrule the formula because they have additional information about the case, but they are wrong more often than not."
- bgilroy26 14y ago
  Just to pitch in 2c, I think society's reticence to implement automated decision making has much more to do with how to handle responsibility for mistakes. I think that is ultimately more important to people than picking the decoder with the best success rate. By distributing responsibility over many doctors (who hail from many medical schools), you make the pain more manageable. The concentrated responsibility that would result from having a single decision system make many many care decisions might be very difficult to stomach.
- DeepDuh 14y ago
  60% out of 200 studies is not all that good, really. It allows the experts to believe that if they're still in the top 25%, no machine can beat them (I know that this is not exactly what the studies say, but statistics are almost always being misinterpreted, so I do it here too). The thing is: I expect Watson to perform significantly better than this. If it can beat Jeopardy grandmasters consistantly and if that technology is correctly being reapplied to the medical field, Watson should easily outperform anything we had before.
  - sesqu 14y ago
    It's worth noting that Watson didn't beat the Jeopardy grandmasters with accuracy, but with speed - not that that isn't useful in healthcare, given sufficient accuracy.
sovande 14y ago
True, so called expert systems are nothing new, but the much improved ability to parse natural language and semantics is newish. It means the human-machine interface is easier to use and combined with portable access via tablets this might very well be the tipping point. Similar to how the iPad was adopted compared to decades of tablets before that never saw a wide-spread adoption, mainly because of the poor human-machine interface.

rtkwe 14y ago

Not all of us. I'm graduating this spring and the top of my fucking awesome hire list was the Watson group in SWG @ IBM.

joonix 14y ago

Mining tweets to determine global reaction to a stadium power outage is far more interesting, of course.