5 ms·

The reproducibility crisis and other problems in science: John Ioannidis [video]

54 points by domofutu 2y ago

d0mine 2y ago
It is worth pointing out that compared to * everything else on the internet, science is the epitome of truth, facts, reproducibility, etc. It might not appear so at the first glance due to Gell-Mann Amnesia effect.
- 7thaccount 2y ago
  I don't think this isn't what the reproducibility issue is about. The problem is there is a broken system in place that rewards publishing in high volumes, only getting positive results, and also doing research that supports those in charge. That doesn't mean new ideas don't win out when clear evidence exists but it is the exception. All of this leads to economic incentives to lie with the data and then you get to scientific papers that people are making decisions based off (like where to put research dollars for Alzheimers) that are fraudulent. This is not good at all.
  - llamaimperative 2y ago
    There are multiple reproducibility crises. The incentive structure is one of them, the other being the general challenge of reproducing results further "up the stack" in highly chaotic fields (the stack being physics -> chem -> bio -> psychology -> sociology, obviously nonexhaustive) I actually find the reproducibility crisis you're mentioning to be both more problematic and to receive far less attention than the other. More problematic because it infects every form of science, even the harder/more fundamental sciences, and because it's way less clear how to fix it. The reproducibility crisis near the top of the stack is just: "do more science and don't believe results until they've been replicated, refined, and matured."
    - maest 2y ago
      Fittingly for this stack model, maths rarely (but not never!) has reproducibility issues. https://xkcd.com/435/ https://xkcd.com/435/
      - llamaimperative 2y ago
        Math is brought up every damn time I bring up this stack! :D
    - 7thaccount 2y ago
      Ah gotcha. Good point. I've always felt that the psychology/sociology studies of the week I hear about on the radio were likely based on some weak statistics in the best of cases and most likely difficult to reproduce.
- karaterobot 2y ago
  Gell-Mann amnesia is when you notice that somebody who is writing about something you're an expert in has made critical mistakes, and should not be trusted, but then you go on and trust people writing about topics you don't know as much about, failing to recognize they're probably also making critical mistakes about those other topics. I'm not sure the connection here. Regarding the comparison between science and folk explanations on the internet, I think it's reasonable to hold science to a higher standard than we hold the general population. If most published research findings are false—as this video claims—and most of what idiots on the internet say is also false, that's neither an equivalence nor a victory for science. On the contrary, it erodes the value of science, both literally and figuratively. Right now we need sources of authority.
  - d0mine 2y ago
    It is not black or white. Yes, there are issues. No, it doesn’t mean a random comment on the internet should have as much authority as scientific paper. There is a strong background of anti-intellectualism right now. The connection with Gell-Mann amnesia is that people overestimate truthiness of what they read online. Combined with reading about [real] issues in science, It might create an impression that scientific findings are relatively at the same level as everything else. My point is even all the troubles science is still a head above despite the perception.
- vacuity 2y ago
  The practice of the scientific method is the path to empirical truth. This doesn't necessarily mean that a nominal scientific finding is true. It is likely true that, on a given topic, an acredited scientist in that field has fairly correct opinions, but I would not take this too far. Conflicts of interest, personal biases, incentives to obtain results, simple lack of reproduction, corrupt peer review, etc. are clearly issues, and it's all the more unfortunate that we can't even say just how deep these issues run. Science is timeless and powerful. Scientists are human beings that nominally, preferably, fallibly practice science.
jMyles 2y ago
This paper was life-changing for me as an undergrad (and I didn't discover the rest of his body of work until I ran into it later here on HN in 2010 or so). We are blessed as a species that John stuck to his principles - and his thirst for empiricism - during the COVID-19 panic, and supported / encouraged his colleagues to do likewise. This video is only the first 12 minutes of the talk. The rest is here (though it is possibly semi-paywalled? It let me watch it, even though it said it was going to make me sign up for a trial): https://iai.tv/video/why-most-published-research-findings-are-false-john-ioannidis?utm_source=YouTube&utm_medium=description https://iai.tv/video/why-most-published-research-findings-ar...
- sdenton4 2y ago
  During the early pandemic, he underestimated the rate of fatality from COVID (which, remember, was much higher before vaccines and paxlovid were deployed), and forcefully advocated for policy based on his lower fatality rate estimates. It was a stunning display of hubris: working on very limited information, he was pushing incautious policy responses which could have cost millions of lives. https://en.m.wikipedia.org/wiki/John_Ioannidis https://en.m.wikipedia.org/wiki/John_Ioannidis
  - DAGdug 2y ago
    As someone who felt the policy reaction to COVID was poor (no balanced assessment of the cost of false positives and negatives in decision-making, poor accounting of uncertainty), I concur that he didn’t apply his usual rigor or his critiques to his own work. He also had, IIRC, a conflict of interest and was funded by the airline industry for this research.
  - cempaka 2y ago
    It's funny that the wildly overestimated IFRs like 3.4%, which were taken as gospel in the early days of the pandemic and drove the actual policy of sweeping shutdowns of schools, preventive medical care, and the economy at large -- a totally experimental and unprecedented measure with no empirical evidence whatsoever to demonstrate the benefits would outweigh the costs -- are not subjected to this same label of "hubris". Iaonnidis's estimates of IFR in the 0.1% to 0.2% range were much closer to the mark.
dgfitz 2y ago
I spent two days trying to get repeatable data from a piece of hardware. Was not able to replicate a dataset once, found out we had a bad high-pressure hose for the brake system and the numbers were all broken. I have no fucking clue how a soft science study could replicate itself.
- stogot 2y ago
  If even an author can’t replicate it, shouldn’t that make the results meaningless?
  - avs733 2y ago
    not neccesarily? It depends on what the claims are. If prior art says 'this is impossible' then a study with one result that is hard to replicate is an interesting insight about the margins. If it is commonly accepted wisdom, we may need to address what that is. If it is your experiment, it doesn't mean your result is wrong but it means it should be significantly questioned. If I say its more that is on me...if I say it is what it is then the results are the results. I think it is a growing, and problematic, misconception that science is about just reporting facts. It is about adding knowledge...soemtimes that is new facts, but more often it is theories, observations, and questions increasingly at the margins of randomness and probability as our species grows and learns.
    - inglor_cz 2y ago
      Substack psychologist Adam Mastroianni argued that if you need advanced statistics to tease out some effect in the soft sciences, it is actually likely that the effect is bogus. How many of those non-reproducible studies showed a robust effect, in contrast to the barely there effect?
    - aeim 2y ago
      so, measuring progress by lines of knowledge added? speaking openly that kinda science seems like a bullshit factory (a very lucrative one, of course) and analogous to an llm hallucination — shallow satisfaction of constraints by enumerating a problem space irrespective of contextual appropriateness, a liability, and forgotten as soon as real value comes along surely the best science isn't just reproducible by scientists — engineers can get the grubby mits on it, to improve (ok, shape) the circumstances of our lives, and survival that other stuff is busywork, not "literature". they are very different things...
bjourne 2y ago
Did no one watch the talk? :) After 13 minutes it ends and tells you to go to a IAI website to continue to watch. Fucking infuriating.
- dang 2y ago
  Yeah that's bad. And https://iai.tv/video/why-most-published-research-findings-are-false-john-ioannidis https://iai.tv/video/why-most-published-research-findings-ar... appears to do the same thing after 30 minutes. Does anyone have a link to the whole thing?
  - moomin 2y ago
    I think it’s fair to say that if someone is that determined for you to not hear what they have to say, you should oblige them.
    - dang 2y ago
      I'm sure Ioannidis isn't the person gatewalling his talk.
domofutu 2y ago
There's also all the (potentially relevant) factors no one's considering - https://www.thetransmitter.org/methods/mouse-housing-temperatures-can-cook-experimental-outcomes/ https://www.thetransmitter.org/methods/mouse-housing-tempera...
- aaron695 2y ago
  A bunch of women going on about giving mice air-conditioning or not? Probably. We've filled science with Imposters. It seems predominantly women, they are the largest change in science. The worst non-replicable sciences are also female majority. Busywork on mice conditions is exactly the false science we are talking about. More excuse why it's not reproducible when mice are mice and are great sub-ins for humans. "In mice" is the meme create to excuse the garbage that's produced. The studies never originally worked on mice, but now they have an excuse why they don't work on humans. I wouldn't necessarily blame it 100% on women though. But I see your point.
mapt 2y ago
This guy doesn't get to say shit to us about literally anything after what he did. 2020's John Ioannidis stood on the shoulders of giants (2005's John Ioannidis) and was one of the first "Experts" to proudly proclaim that in his professional judgement, COVID-19 was no big deal and didn't warrant countermeasures. https://sciencebasedmedicine.org/mistakes/ https://sciencebasedmedicine.org/mistakes/ If you choose to become a policy entrepreneur and your success ends in a Holocaust-scale outcome, you get to shut up, retire, and thank your lucky stars we live in a society that strongly discourages blood debts. That's the deal. No book tour, do not pass go, do not collect $200, and stay the fuck off social media.
- ttoinou 2y ago
  Seems like he was correct though. This article doesn’t debunk him, you can’t use official statistics to debunk a claim that the statistics are not computed correctly
  - deleted 2y ago
    [deleted]
- decUser3 2y ago
  > in an effort— well intentioned—to control the coronavirus, we may inflict great damage on ourselves. And you believe he was wrong and you believe Covid was a "Holocaust-scale outcome" ? He said the rates would be exaggerated and now with the benefit of hindsight, although some would claim common sense after seeing the sensationalism of the media, we can see he was correct. Or do you believe that's not true ?

54 points by domofutu 2y ago

It is worth pointing out that compared to * everything else on the internet, science is the epitome of truth, facts, reproducibility, etc. It might not appear so at the first glance due to Gell-Mann Amnesia effect.

7thaccount 2y ago
I don't think this isn't what the reproducibility issue is about. The problem is there is a broken system in place that rewards publishing in high volumes, only getting positive results, and also doing research that supports those in charge. That doesn't mean new ideas don't win out when clear evidence exists but it is the exception. All of this leads to economic incentives to lie with the data and then you get to scientific papers that people are making decisions based off (like where to put research dollars for Alzheimers) that are fraudulent. This is not good at all.
- llamaimperative 2y ago
  There are multiple reproducibility crises. The incentive structure is one of them, the other being the general challenge of reproducing results further "up the stack" in highly chaotic fields (the stack being physics -> chem -> bio -> psychology -> sociology, obviously nonexhaustive) I actually find the reproducibility crisis you're mentioning to be both more problematic and to receive far less attention than the other. More problematic because it infects every form of science, even the harder/more fundamental sciences, and because it's way less clear how to fix it. The reproducibility crisis near the top of the stack is just: "do more science and don't believe results until they've been replicated, refined, and matured."
  - maest 2y ago
    Fittingly for this stack model, maths rarely (but not never!) has reproducibility issues. https://xkcd.com/435/ https://xkcd.com/435/
    - llamaimperative 2y ago
      Math is brought up every damn time I bring up this stack! :D
  - 7thaccount 2y ago
    Ah gotcha. Good point. I've always felt that the psychology/sociology studies of the week I hear about on the radio were likely based on some weak statistics in the best of cases and most likely difficult to reproduce.
karaterobot 2y ago
Gell-Mann amnesia is when you notice that somebody who is writing about something you're an expert in has made critical mistakes, and should not be trusted, but then you go on and trust people writing about topics you don't know as much about, failing to recognize they're probably also making critical mistakes about those other topics. I'm not sure the connection here. Regarding the comparison between science and folk explanations on the internet, I think it's reasonable to hold science to a higher standard than we hold the general population. If most published research findings are false—as this video claims—and most of what idiots on the internet say is also false, that's neither an equivalence nor a victory for science. On the contrary, it erodes the value of science, both literally and figuratively. Right now we need sources of authority.
- d0mine 2y ago
  It is not black or white. Yes, there are issues. No, it doesn’t mean a random comment on the internet should have as much authority as scientific paper. There is a strong background of anti-intellectualism right now. The connection with Gell-Mann amnesia is that people overestimate truthiness of what they read online. Combined with reading about [real] issues in science, It might create an impression that scientific findings are relatively at the same level as everything else. My point is even all the troubles science is still a head above despite the perception.
vacuity 2y ago
The practice of the scientific method is the path to empirical truth. This doesn't necessarily mean that a nominal scientific finding is true. It is likely true that, on a given topic, an acredited scientist in that field has fairly correct opinions, but I would not take this too far. Conflicts of interest, personal biases, incentives to obtain results, simple lack of reproduction, corrupt peer review, etc. are clearly issues, and it's all the more unfortunate that we can't even say just how deep these issues run. Science is timeless and powerful. Scientists are human beings that nominally, preferably, fallibly practice science.

jMyles 2y ago

This paper was life-changing for me as an undergrad (and I didn't discover the rest of his body of work until I ran into it later here on HN in 2010 or so). We are blessed as a species that John stuck to his principles - and his thirst for empiricism - during the COVID-19 panic, and supported / encouraged his colleagues to do likewise. This video is only the first 12 minutes of the talk. The rest is here (though it is possibly semi-paywalled? It let me watch it, even though it said it was going to make me sign up for a trial): https://iai.tv/video/why-most-published-research-findings-are-false-john-ioannidis?utm_source=YouTube&utm_medium=description https://iai.tv/video/why-most-published-research-findings-ar...

sdenton4 2y ago
During the early pandemic, he underestimated the rate of fatality from COVID (which, remember, was much higher before vaccines and paxlovid were deployed), and forcefully advocated for policy based on his lower fatality rate estimates. It was a stunning display of hubris: working on very limited information, he was pushing incautious policy responses which could have cost millions of lives. https://en.m.wikipedia.org/wiki/John_Ioannidis https://en.m.wikipedia.org/wiki/John_Ioannidis
- DAGdug 2y ago
  As someone who felt the policy reaction to COVID was poor (no balanced assessment of the cost of false positives and negatives in decision-making, poor accounting of uncertainty), I concur that he didn’t apply his usual rigor or his critiques to his own work. He also had, IIRC, a conflict of interest and was funded by the airline industry for this research.
- cempaka 2y ago
  It's funny that the wildly overestimated IFRs like 3.4%, which were taken as gospel in the early days of the pandemic and drove the actual policy of sweeping shutdowns of schools, preventive medical care, and the economy at large -- a totally experimental and unprecedented measure with no empirical evidence whatsoever to demonstrate the benefits would outweigh the costs -- are not subjected to this same label of "hubris". Iaonnidis's estimates of IFR in the 0.1% to 0.2% range were much closer to the mark.

dgfitz 2y ago

I spent two days trying to get repeatable data from a piece of hardware. Was not able to replicate a dataset once, found out we had a bad high-pressure hose for the brake system and the numbers were all broken. I have no fucking clue how a soft science study could replicate itself.

stogot 2y ago
If even an author can’t replicate it, shouldn’t that make the results meaningless?
- avs733 2y ago
  not neccesarily? It depends on what the claims are. If prior art says 'this is impossible' then a study with one result that is hard to replicate is an interesting insight about the margins. If it is commonly accepted wisdom, we may need to address what that is. If it is your experiment, it doesn't mean your result is wrong but it means it should be significantly questioned. If I say its more that is on me...if I say it is what it is then the results are the results. I think it is a growing, and problematic, misconception that science is about just reporting facts. It is about adding knowledge...soemtimes that is new facts, but more often it is theories, observations, and questions increasingly at the margins of randomness and probability as our species grows and learns.
  - inglor_cz 2y ago
    Substack psychologist Adam Mastroianni argued that if you need advanced statistics to tease out some effect in the soft sciences, it is actually likely that the effect is bogus. How many of those non-reproducible studies showed a robust effect, in contrast to the barely there effect?
  - aeim 2y ago
    so, measuring progress by lines of knowledge added? speaking openly that kinda science seems like a bullshit factory (a very lucrative one, of course) and analogous to an llm hallucination — shallow satisfaction of constraints by enumerating a problem space irrespective of contextual appropriateness, a liability, and forgotten as soon as real value comes along surely the best science isn't just reproducible by scientists — engineers can get the grubby mits on it, to improve (ok, shape) the circumstances of our lives, and survival that other stuff is busywork, not "literature". they are very different things...

bjourne 2y ago

Did no one watch the talk? :) After 13 minutes it ends and tells you to go to a IAI website to continue to watch. Fucking infuriating.

dang 2y ago
Yeah that's bad. And https://iai.tv/video/why-most-published-research-findings-are-false-john-ioannidis https://iai.tv/video/why-most-published-research-findings-ar... appears to do the same thing after 30 minutes. Does anyone have a link to the whole thing?
- moomin 2y ago
  I think it’s fair to say that if someone is that determined for you to not hear what they have to say, you should oblige them.
  - dang 2y ago
    I'm sure Ioannidis isn't the person gatewalling his talk.

domofutu 2y ago

There's also all the (potentially relevant) factors no one's considering - https://www.thetransmitter.org/methods/mouse-housing-temperatures-can-cook-experimental-outcomes/ https://www.thetransmitter.org/methods/mouse-housing-tempera...

aaron695 2y ago
A bunch of women going on about giving mice air-conditioning or not? Probably. We've filled science with Imposters. It seems predominantly women, they are the largest change in science. The worst non-replicable sciences are also female majority. Busywork on mice conditions is exactly the false science we are talking about. More excuse why it's not reproducible when mice are mice and are great sub-ins for humans. "In mice" is the meme create to excuse the garbage that's produced. The studies never originally worked on mice, but now they have an excuse why they don't work on humans. I wouldn't necessarily blame it 100% on women though. But I see your point.

mapt 2y ago

This guy doesn't get to say shit to us about literally anything after what he did. 2020's John Ioannidis stood on the shoulders of giants (2005's John Ioannidis) and was one of the first "Experts" to proudly proclaim that in his professional judgement, COVID-19 was no big deal and didn't warrant countermeasures. https://sciencebasedmedicine.org/mistakes/ https://sciencebasedmedicine.org/mistakes/ If you choose to become a policy entrepreneur and your success ends in a Holocaust-scale outcome, you get to shut up, retire, and thank your lucky stars we live in a society that strongly discourages blood debts. That's the deal. No book tour, do not pass go, do not collect $200, and stay the fuck off social media.

ttoinou 2y ago
Seems like he was correct though. This article doesn’t debunk him, you can’t use official statistics to debunk a claim that the statistics are not computed correctly
- deleted 2y ago
  [deleted]
decUser3 2y ago
> in an effort— well intentioned—to control the coronavirus, we may inflict great damage on ourselves. And you believe he was wrong and you believe Covid was a "Holocaust-scale outcome" ? He said the rates would be exaggerated and now with the benefit of hindsight, although some would claim common sense after seeing the sensationalism of the media, we can see he was correct. Or do you believe that's not true ?