6 ms·

One of my favourite stories along these lines (this is from Ben Goldacre's great book Bad Science, and may be slightly misremembered in parts) is about Archie C

One of my favourite stories along these lines (this is from Ben Goldacre's great book Bad Science, and may be slightly misremembered in parts) is about Archie Cochrane, who gave his name to the Cochrane Collaboration, an organisation who rigorously analyse the evidence for medical treatments, even "obvious" ones. He set it up after being a Japanese prisoner of war, where he had to make decisions about medical treatment in the face of extremely callous treatment by the Japanese officers. They were often decisions you only hear about in philosophical games - e.g. do you save one person's life or the sight of ten people - and he realised that much of his medical training was not based on proper evidence. In the UK in the 1970s, hospitals started equipping expensive cardiac units, specifically designed for the treatment of heart attacks, full of expensive medical equipment (and if HN doesn't mind a good-natured tease, cardiac surgeons don't exactly shun the heroic mode of medicine). Part-way through a Cochrane study of their effectiveness, the data was pointing to survival rates being slightly higher when people were treated at home rather than in the cardiac units, but the results were not statistically significant. He decided to play a trick and annouced at an interim meeting that the data were pointing the opposite way - that treatment in cardiac units showed slightly higher survival rates, but were still not statistically significant. The doctors in the meeting were extremely angry that he insisted that the study should go on because the results were not statistically significant and therefore no conclusions could be drawn from the data. It wasn't until later in the meeting that he came clean, and told them that the data were pointing in the other direction. Funnily enough, none of the doctors still called for the study to be halted early in the face of identical evidence, just pointing to a different conclusion.

saas_sam 6y ago
This kind of thinking is so common I often wonder how the hell we manage to progress as a species. People seem to only ever look at the evidence that supports positions they originally adopted for bad reasons. I can't remember the last time I talked to anyone who could recall to me a single time they changed their mind based on seriously weighing evidence for & against their positions. People Google to find out why they're right and that's the end of it.
- satellite2 6y ago
  Maybe because we are not really made to adapt as individuals, only as a specie? And that we reached this level of collaboration and specialization, while usual at a cellular level is unknown at the individual level?
  - MrBuddyCasino 6y ago
    I think thats going into the right direction. Rationalism is a dead end on the individual level. Some thoughts in this direction: https://alexkaschuta.substack.com/p/trad-the-final-frontier https://alexkaschuta.substack.com/p/trad-the-final-frontier
    - QuadmasterXLII 6y ago
      The issue with the trad philosophy is that these cute blog posts never clarify what to do when the “haha quirky” traditions passed down have real life victims. It’s one thing to stick to “let the cassava rest” or “don’t eat shellfish” but I’ll be damned before I go along with “stone the rape victims” or “enslave anyone with skin darker than a manilla folder”
      - MrBuddyCasino 6y ago
        This is more about a change in perspective. Its about giving a voice to the generations that existed before us, and what they have to tell us about how societies function and how to optimize for human happiness. The alternative is making up the rules of how civilisation works from first principles using a rationalist blank slate, which went horribly wrong when the communists did it, and goes completely off the rails right now [0]. The idea that we are rational decision-making machines, where all you have to do is to allow maximum freedom of decision making on an individual level, and you will automatically arrive at a happiness maximum, turns out not to be true. Communities are important, (shared) values are important, the right norms are important. There is much of value to be learned. It is not about accepting all the old ideas, because norms are evolving for good reasons, but instead take a hard look and evaluate which ones actually made a lot of sense. [0] https://twitter.com/swipewright/status/1340199529891549184?s=21 https://twitter.com/swipewright/status/1340199529891549184?s...
Alex3917 6y ago
I mean Ben Goldacre's book itself is a pretty good example of people being irrational. He's mostly just covering the same ground as Marcia Angell's earlier book The Truth About Drug Companies, which "skeptics" hated. But when Goldacre published his book the same folks decided they loved it, presumably because it was written by a man. I mean don't get me wrong, there's enough difference between them that it's absolutely worth reading both of them, but the difference in reception between them was shocking.
- Siira 6y ago
  It’s toxic to just assume/accuse sexism out of thin air.
- Jedd 6y ago
  I mean ... parent was talking about Bad Science, Ben's first book in 2008. You seem to be comparing his second book, Bad Pharma (2012) with Marcia Angell's 'Truth About the Drug Companies' (2005), and assuming some claimed disparity in popularity is down to author gender. It may be that new readers will pick a more recent publication, assuming it to be more contemporary. Or that they're (more) familiar with someone that had generated a large online following. (I think that's why I picked up his book, but had not heard of Marcia Angell.) OTOH I'm not sure what '"skeptics" hated' refers to. Looking now on Amazon (I am not happy about visiting this site) both are showing 4.6 / 5.0 ratings. On Goodreads Bad Pharma is showing 4.1 compared to Truth with 3.95 (but that's a pretty tight race).
rscho 6y ago
The Cochrane Collaboration is a noble effort but its impact is limited in practice. There is a huge incentive to produce papers in the modern medical system, and I'm in a good place to know that you can make the data say whatever you want with techniques such as meta analysis, which incidentally is a big part of Cochrane studies. Meta analysis as a field has grown trendemously because who doesn't want to publish high impact papers without having to gather any new data, right? Publication of end-to-end single-command build reproducible research should be the only one allowed at this point, with everything accessible to the reader, including data. I've been doing shit medical science for ten years already because of hierarchical pressure, and I'd love a change of course.
- Tenoke 6y ago
  As far as I can tell the Cochrane meta-analyses are usually of above average quality, pretty honest, don't have particular reason or evidence for bias in any direction and are conducted fairly well. Do you have any specific examples of problems in their meta-analyses?
  - rscho 6y ago
    Firstly, I have a problem with meta analysis in general and the sheer volume of meta papers tends to drown the Cochrane papers. Additionally, I challenge the judgment calls about honesty of authors, be it the source trial authors or the meta authors. There is a _really_ large interpretation gap between what is written and what really happened depending on how you encode the data into the dataset and the analysis model. I am a clinical researcher. And TBH, I think it's not really possible to conduct truthful and fundamentally believable clinical research given the current social and technological environments of academic research. I'm far from the only one slaving for professors who don't know what they're doing. If I did not have a family to feed, I would not accept doing the work I am doing right now.
- neuronic 6y ago
  Meta-analysis is still very valuable though because you would have a hard time finding significant disagreements between studies otherwise. Of course it comes with the benefit of not having to produce novel data but IMO that does not absolve such studies from merit and impact.
appleflaxen 6y ago
I don't follow your anecdote. Why would the surgeons want a non-significant study to stop when they were coming out on top, and want it to continue when they were coming out on the bottom? In either case it's non-significant, so continuing it will likely 1) continue to be negative, or 2) develop large enough n that the trend becomes signifant. If it's 2, then the incentives are exactly the opposite of what you outlined, if I'm understanding what you wrote.
littlestymaar 6y ago
Nitpick: Cochrane was British then did not fight against the Japanese, and he was prisoner of war in a German camp. Here is his story about the prisoners' camp[1] and there is no mention of blinding people, “just” German soldiers shooting or throwing grenades at random on prisoners. [1]: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1444794/?page=1 https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1444794/?page=1
- frereubu 6y ago
  Fair nitpick - as I said, I was recalling the book from memory having read it quite a while ago, and couldn't remember what the exact tradeoffs were. Thanks for linking to an original source.

saas_sam 6y ago

This kind of thinking is so common I often wonder how the hell we manage to progress as a species. People seem to only ever look at the evidence that supports positions they originally adopted for bad reasons. I can't remember the last time I talked to anyone who could recall to me a single time they changed their mind based on seriously weighing evidence for & against their positions. People Google to find out why they're right and that's the end of it.

satellite2 6y ago
Maybe because we are not really made to adapt as individuals, only as a specie? And that we reached this level of collaboration and specialization, while usual at a cellular level is unknown at the individual level?
- MrBuddyCasino 6y ago
  I think thats going into the right direction. Rationalism is a dead end on the individual level. Some thoughts in this direction: https://alexkaschuta.substack.com/p/trad-the-final-frontier https://alexkaschuta.substack.com/p/trad-the-final-frontier
  - QuadmasterXLII 6y ago
    The issue with the trad philosophy is that these cute blog posts never clarify what to do when the “haha quirky” traditions passed down have real life victims. It’s one thing to stick to “let the cassava rest” or “don’t eat shellfish” but I’ll be damned before I go along with “stone the rape victims” or “enslave anyone with skin darker than a manilla folder”
    - MrBuddyCasino 6y ago
      This is more about a change in perspective. Its about giving a voice to the generations that existed before us, and what they have to tell us about how societies function and how to optimize for human happiness. The alternative is making up the rules of how civilisation works from first principles using a rationalist blank slate, which went horribly wrong when the communists did it, and goes completely off the rails right now [0]. The idea that we are rational decision-making machines, where all you have to do is to allow maximum freedom of decision making on an individual level, and you will automatically arrive at a happiness maximum, turns out not to be true. Communities are important, (shared) values are important, the right norms are important. There is much of value to be learned. It is not about accepting all the old ideas, because norms are evolving for good reasons, but instead take a hard look and evaluate which ones actually made a lot of sense. [0] https://twitter.com/swipewright/status/1340199529891549184?s=21 https://twitter.com/swipewright/status/1340199529891549184?s...

Alex3917 6y ago

I mean Ben Goldacre's book itself is a pretty good example of people being irrational. He's mostly just covering the same ground as Marcia Angell's earlier book The Truth About Drug Companies, which "skeptics" hated. But when Goldacre published his book the same folks decided they loved it, presumably because it was written by a man. I mean don't get me wrong, there's enough difference between them that it's absolutely worth reading both of them, but the difference in reception between them was shocking.

Siira 6y ago
It’s toxic to just assume/accuse sexism out of thin air.
Jedd 6y ago
I mean ... parent was talking about Bad Science, Ben's first book in 2008. You seem to be comparing his second book, Bad Pharma (2012) with Marcia Angell's 'Truth About the Drug Companies' (2005), and assuming some claimed disparity in popularity is down to author gender. It may be that new readers will pick a more recent publication, assuming it to be more contemporary. Or that they're (more) familiar with someone that had generated a large online following. (I think that's why I picked up his book, but had not heard of Marcia Angell.) OTOH I'm not sure what '"skeptics" hated' refers to. Looking now on Amazon (I am not happy about visiting this site) both are showing 4.6 / 5.0 ratings. On Goodreads Bad Pharma is showing 4.1 compared to Truth with 3.95 (but that's a pretty tight race).

rscho 6y ago

The Cochrane Collaboration is a noble effort but its impact is limited in practice. There is a huge incentive to produce papers in the modern medical system, and I'm in a good place to know that you can make the data say whatever you want with techniques such as meta analysis, which incidentally is a big part of Cochrane studies. Meta analysis as a field has grown trendemously because who doesn't want to publish high impact papers without having to gather any new data, right? Publication of end-to-end single-command build reproducible research should be the only one allowed at this point, with everything accessible to the reader, including data. I've been doing shit medical science for ten years already because of hierarchical pressure, and I'd love a change of course.

Tenoke 6y ago
As far as I can tell the Cochrane meta-analyses are usually of above average quality, pretty honest, don't have particular reason or evidence for bias in any direction and are conducted fairly well. Do you have any specific examples of problems in their meta-analyses?
- rscho 6y ago
  Firstly, I have a problem with meta analysis in general and the sheer volume of meta papers tends to drown the Cochrane papers. Additionally, I challenge the judgment calls about honesty of authors, be it the source trial authors or the meta authors. There is a _really_ large interpretation gap between what is written and what really happened depending on how you encode the data into the dataset and the analysis model. I am a clinical researcher. And TBH, I think it's not really possible to conduct truthful and fundamentally believable clinical research given the current social and technological environments of academic research. I'm far from the only one slaving for professors who don't know what they're doing. If I did not have a family to feed, I would not accept doing the work I am doing right now.
neuronic 6y ago
Meta-analysis is still very valuable though because you would have a hard time finding significant disagreements between studies otherwise. Of course it comes with the benefit of not having to produce novel data but IMO that does not absolve such studies from merit and impact.

appleflaxen 6y ago

I don't follow your anecdote. Why would the surgeons want a non-significant study to stop when they were coming out on top, and want it to continue when they were coming out on the bottom? In either case it's non-significant, so continuing it will likely 1) continue to be negative, or 2) develop large enough n that the trend becomes signifant. If it's 2, then the incentives are exactly the opposite of what you outlined, if I'm understanding what you wrote.

littlestymaar 6y ago

Nitpick: Cochrane was British then did not fight against the Japanese, and he was prisoner of war in a German camp. Here is his story about the prisoners' camp[1] and there is no mention of blinding people, “just” German soldiers shooting or throwing grenades at random on prisoners. [1]: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1444794/?page=1 https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1444794/?page=1

frereubu 6y ago
Fair nitpick - as I said, I was recalling the book from memory having read it quite a while ago, and couldn't remember what the exact tradeoffs were. Thanks for linking to an original source.