6 ms·

It's worth noting that humans completely suck at almost any task even simple statistics can be applied to. Even world class experts will typically be beat by si

It's worth noting that humans completely suck at almost any task even simple statistics can be applied to. Even world class experts will typically be beat by simple linear regression. See here for many examples: http://lesswrong.com/lw/3gv/statistical_prediction_rules_outperform_expert/ http://lesswrong.com/lw/3gv/statistical_prediction_rules_out... And this is very old research with methods that are very primitive compared to modern machine learning (most of this was done on pencil and paper!): >Wittman (1941) constructed an SPR that predicted the success of electroshock therapy for patients more reliably than the medical or psychological staff. >Carroll et. al. (1988) found an SPR that predicts criminal recidivism better than expert criminologists. >An SPR constructed by Goldberg (1968) did a better job of diagnosing patients as neurotic or psychotic than did trained clinical psychologists. It's completely amazing we allow human doctors to make diagnoses at all. At the very least algorithms should always be part of the process (but note that humans given the results of an algorithm still do worse than just the algorithm on it's own.) The problem is there has been enormous resistance to the use of algorithms in medicine. People irrationally distrust algorithms and strongly prefer human judgement. Even when they know the algorithms are superior. Psychologists have actually studied this phenomenon and have named it "Algorithm Aversion": http://opim.wharton.upenn.edu/risk/library/WPAF201410-AlgorthimAversion-Dietvorst-Simmons-Massey.pdf http://opim.wharton.upenn.edu/risk/library/WPAF201410-Algort... This isn't even getting into the institutional resistance to change or having people lose their jobs to robots.

carbocation 10y ago
Is there evidence that there are high-performing algorithms that are available to doctors that they aren't using? The basic algorithms like RCRI and CHADS2VASC are almost universally applied in the appropriate context (at my quaternary care teaching hospital, granted). If there are numerous algorithms that we should be using but aren't, I'd argue that the bigger barrier is making them accessible and easy enough to use that they can be applied during a clinical encounter (among all the other things that need to be done).
- gleb 10y ago
  Here's a simple algorithm that would help millions, yet it is not accessible to most patients: Ordering A1C, remembering to actually check the results, and telling patient to stop eating sugar - well before they are symptomatic for diabetes.
  - chimeracoder 10y ago
    > Ordering A1C, remembering to actually check the results, and telling patient to stop eating sugar - well before they are symptomatic for diabetes. Yeah, it's that last part ("telling patient to stop eating sugar" [and getting them to actually change their behavior]) that's the hard part. This is the holy grail of medicine and has been for decades. We've done billions of dollars of research over the last century, and the one clear thing we've found is that we do not know of any cost-effective way to get patients to change their behavior at large[0]. Seriously, if you really have the answer to that question, you could be a multi-billionaire. [0] That is, we can change behavior of some patients, but they're generally the patients who would have changed their behavior anyway. And we can change the behavior of some of the rest, but they're not cost-effective.
    - TeMPOraL 10y ago
      If you knew how to do that, you could also solve the climate change / global warming problem, as well as host of other big social issues at the same time. As it is, people simply follow the incentive gradient, and most of the time there's damn little you can do about it.
      - neffy 10y ago
        And so it turns out once again, in every field, people really are the problem ...
      - lliamander 10y ago
        There is, of course, the (unintended?) side-effect that if there was a cost-effective way for controlling peoples' behavior then we would be effectively under the tyranny of whomever possessed those means. Sometimes we just have to let other people live with the consequences of their actions; which is hard, because often we also have to live with those consequences. Whatever we say about "other people", we should also remember that we are "other people" to somebody.
stewbrew 10y ago
> The problem is there has been enormous resistance to the use of algorithms in medicine That's not 100% true for all disciplines. E.g., in radiology, there are some assisting technologies around for identifying tumors etc. While many radiologists use them daily, the evidence isn't always that clear. Often they just lead to a higher false positive rate.
SubiculumCode 10y ago
I've spent a lot of time doing manual tracings of brain structures and I tell you now that while automated methods are improving rapidly, and just now approaching the average ability of a tracer, they do not rival experts yet. The most time efficient methods now is semi-automatic. Begin with the automated methods, then make manual corrections. Given all that, brain radiology is a lot easier that pathological radiology (e.g. tracing the extent of a lung tumor). There is a lot of research into automating this, but dice overlaps are still poor.
rayiner 10y ago
It's interesting that you mention the criminology scenario. Courts are starting to use software algorithms to predict recidivism rates for purposes of setting bond amounts. The reports are even given to judges to inform sentencing. However, there is an enormous backlash against them, because e.g. they predict higher rates for racial minorities: https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing https://www.propublica.org/article/machine-bias-risk-assessm....
- Houshalter 10y ago
  The propublica study has been shown to be a lie, see https://www.chrisstucchio.com/blog/2016/propublica_is_lying.html https://www.chrisstucchio.com/blog/2016/propublica_is_lying.... Anyway all I'm claiming is that statistical methods are much more accurate than humans. Nothing there disputes that claim. If you want the most accurate predictions possible, you should use an algorithm. That article implies that humans are somehow fair or unbiased. That is a completely ridiculous claim that has been proven false many times. Human judges give ugly people twice the sentences of attractive people. Judges have been shown to give significantly harsher sentences just before lunch, when they are hungry. Not to mention all the classic biases against gender/race/political affiliation/etc. Studies have shown interviews are worse than useless at assessing how good someone will be as an employee. Instead employers are biased by how much they like the candidate. We should hardly expect traditional parole interviews to be any different. But almost no one cares about these results. Yet when an algorithm is shown to have a (not statistically significant) bias, people freak out. This, if anything, proves my point that algorithm aversion is a serious problem.
  - scott_s 10y ago
    Chris Stucchio's point is different from that of the article. The problem is that the word "bias" is overloaded, and used in two different ways. When ProPublica uses the word bias, they mean that one race is affected in a systematically different way than another race. When Chris Stucchio uses the word bias, he means a statistical method which has a built in bias to achieve a particular result. It is quite possible for both to be true: the statistical methods have no built in bias, but their prediction ends up making different predictions for different races. I find the most reasonable explanation to be that society itself has a racial bias, and society itself is the data generator.

carbocation 10y ago

Is there evidence that there are high-performing algorithms that are available to doctors that they aren't using? The basic algorithms like RCRI and CHADS2VASC are almost universally applied in the appropriate context (at my quaternary care teaching hospital, granted). If there are numerous algorithms that we should be using but aren't, I'd argue that the bigger barrier is making them accessible and easy enough to use that they can be applied during a clinical encounter (among all the other things that need to be done).

gleb 10y ago
Here's a simple algorithm that would help millions, yet it is not accessible to most patients: Ordering A1C, remembering to actually check the results, and telling patient to stop eating sugar - well before they are symptomatic for diabetes.
- chimeracoder 10y ago
  > Ordering A1C, remembering to actually check the results, and telling patient to stop eating sugar - well before they are symptomatic for diabetes. Yeah, it's that last part ("telling patient to stop eating sugar" [and getting them to actually change their behavior]) that's the hard part. This is the holy grail of medicine and has been for decades. We've done billions of dollars of research over the last century, and the one clear thing we've found is that we do not know of any cost-effective way to get patients to change their behavior at large[0]. Seriously, if you really have the answer to that question, you could be a multi-billionaire. [0] That is, we can change behavior of some patients, but they're generally the patients who would have changed their behavior anyway. And we can change the behavior of some of the rest, but they're not cost-effective.
  - TeMPOraL 10y ago
    If you knew how to do that, you could also solve the climate change / global warming problem, as well as host of other big social issues at the same time. As it is, people simply follow the incentive gradient, and most of the time there's damn little you can do about it.
    - neffy 10y ago
      And so it turns out once again, in every field, people really are the problem ...
    - lliamander 10y ago
      There is, of course, the (unintended?) side-effect that if there was a cost-effective way for controlling peoples' behavior then we would be effectively under the tyranny of whomever possessed those means. Sometimes we just have to let other people live with the consequences of their actions; which is hard, because often we also have to live with those consequences. Whatever we say about "other people", we should also remember that we are "other people" to somebody.

stewbrew 10y ago

> The problem is there has been enormous resistance to the use of algorithms in medicine That's not 100% true for all disciplines. E.g., in radiology, there are some assisting technologies around for identifying tumors etc. While many radiologists use them daily, the evidence isn't always that clear. Often they just lead to a higher false positive rate.

SubiculumCode 10y ago

I've spent a lot of time doing manual tracings of brain structures and I tell you now that while automated methods are improving rapidly, and just now approaching the average ability of a tracer, they do not rival experts yet. The most time efficient methods now is semi-automatic. Begin with the automated methods, then make manual corrections. Given all that, brain radiology is a lot easier that pathological radiology (e.g. tracing the extent of a lung tumor). There is a lot of research into automating this, but dice overlaps are still poor.

rayiner 10y ago

It's interesting that you mention the criminology scenario. Courts are starting to use software algorithms to predict recidivism rates for purposes of setting bond amounts. The reports are even given to judges to inform sentencing. However, there is an enormous backlash against them, because e.g. they predict higher rates for racial minorities: https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing https://www.propublica.org/article/machine-bias-risk-assessm....

Houshalter 10y ago
The propublica study has been shown to be a lie, see https://www.chrisstucchio.com/blog/2016/propublica_is_lying.html https://www.chrisstucchio.com/blog/2016/propublica_is_lying.... Anyway all I'm claiming is that statistical methods are much more accurate than humans. Nothing there disputes that claim. If you want the most accurate predictions possible, you should use an algorithm. That article implies that humans are somehow fair or unbiased. That is a completely ridiculous claim that has been proven false many times. Human judges give ugly people twice the sentences of attractive people. Judges have been shown to give significantly harsher sentences just before lunch, when they are hungry. Not to mention all the classic biases against gender/race/political affiliation/etc. Studies have shown interviews are worse than useless at assessing how good someone will be as an employee. Instead employers are biased by how much they like the candidate. We should hardly expect traditional parole interviews to be any different. But almost no one cares about these results. Yet when an algorithm is shown to have a (not statistically significant) bias, people freak out. This, if anything, proves my point that algorithm aversion is a serious problem.
- scott_s 10y ago
  Chris Stucchio's point is different from that of the article. The problem is that the word "bias" is overloaded, and used in two different ways. When ProPublica uses the word bias, they mean that one race is affected in a systematically different way than another race. When Chris Stucchio uses the word bias, he means a statistical method which has a built in bias to achieve a particular result. It is quite possible for both to be true: the statistical methods have no built in bias, but their prediction ends up making different predictions for different races. I find the most reasonable explanation to be that society itself has a racial bias, and society itself is the data generator.