3 ms·

The book "Thinking, Fast and Slow" has a great chapter on human predictions versus those made by forumlas. The superiority of simple forumlas was being demonstr

The book "Thinking, Fast and Slow" has a great chapter on human predictions versus those made by forumlas. The superiority of simple forumlas was being demonstrated way back in the 50s (see "Clinical Versus Statistical Prediction: A Theoretical Analysis And A Review Of The Evidence"). Since then the evidence for statistical rather than intuitive prediction has only become stronger. From "Thinking, Fast and Slow": "The number of studies reporting comparisons of clinical and statistical predictions has increased to roughly two hundred, but the score in the contest between algorithms and humans has not changed. About 60% of the studies have shown significantly better accuracy for the algorithms. The other comparisons scored a draw in accuracy, but a tie is tantamount to a win for the statistical rules, which are normally much less expensive to use than expert judgment. No exception has been convincingly documented." Also interesting: "Several studies have shown that human decision makers are inferior to a prediction formula even when they are given the score suggested by the formula! They feel that they can overrule the formula because they have additional information about the case, but they are wrong more often than not."

bgilroy26 14y ago
Just to pitch in 2c, I think society's reticence to implement automated decision making has much more to do with how to handle responsibility for mistakes. I think that is ultimately more important to people than picking the decoder with the best success rate. By distributing responsibility over many doctors (who hail from many medical schools), you make the pain more manageable. The concentrated responsibility that would result from having a single decision system make many many care decisions might be very difficult to stomach.
DeepDuh 14y ago
60% out of 200 studies is not all that good, really. It allows the experts to believe that if they're still in the top 25%, no machine can beat them (I know that this is not exactly what the studies say, but statistics are almost always being misinterpreted, so I do it here too). The thing is: I expect Watson to perform significantly better than this. If it can beat Jeopardy grandmasters consistantly and if that technology is correctly being reapplied to the medical field, Watson should easily outperform anything we had before.
- sesqu 14y ago
  It's worth noting that Watson didn't beat the Jeopardy grandmasters with accuracy, but with speed - not that that isn't useful in healthcare, given sufficient accuracy.

bgilroy26 14y ago

Just to pitch in 2c, I think society's reticence to implement automated decision making has much more to do with how to handle responsibility for mistakes. I think that is ultimately more important to people than picking the decoder with the best success rate. By distributing responsibility over many doctors (who hail from many medical schools), you make the pain more manageable. The concentrated responsibility that would result from having a single decision system make many many care decisions might be very difficult to stomach.

DeepDuh 14y ago

60% out of 200 studies is not all that good, really. It allows the experts to believe that if they're still in the top 25%, no machine can beat them (I know that this is not exactly what the studies say, but statistics are almost always being misinterpreted, so I do it here too). The thing is: I expect Watson to perform significantly better than this. If it can beat Jeopardy grandmasters consistantly and if that technology is correctly being reapplied to the medical field, Watson should easily outperform anything we had before.

sesqu 14y ago
It's worth noting that Watson didn't beat the Jeopardy grandmasters with accuracy, but with speed - not that that isn't useful in healthcare, given sufficient accuracy.