4 ms·

"coronavirus is what happens when you put the reproducibility crisis in charge of the planet"

umvi 6y ago
-- @mwilcox[0] [0] https://twitter.com/mwilcox/status/1260458110596120576 https://twitter.com/mwilcox/status/1260458110596120576
0xcafecafe 6y ago
Can someone please elaborate on this? What is the "reproducibility crisis" in this case?
- xapata 6y ago
  "Most published research findings are false" (not reproducible) https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1182327/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1182327/
  - whydoyoucare 6y ago
    This is an excellent read: https://slatestarcodex.com/2014/12/12/beware-the-man-of-one-study/ https://slatestarcodex.com/2014/12/12/beware-the-man-of-one-... -- basically says one has to be very skeptical of trusting a study that suits the narrative, and how studies can be cherry-picked to align with your narrative.
- umvi 6y ago
  Reproducibility crisis is well known, especially in social sciences. However, it affects other science fields as well. In a nutshell, the crisis is that there is far more incentive to churn out a new publication/study than reproduce an existing one. So we have tons of studies that are the only one of its kind that have not been reproduced. The crisis part comes in where attempts to reproduce the results of these studies fail which throws into question the legitimacy of the entire field's publications. So you can imagine how putting your faith into covid-19 related studies (such as effects of hydroxychloroquine) that were rushed and results not reproduced independently can result in problems down the road. Science is probably the best tool we have for determining truth. However, it takes a long time and a lot of effort to converge on the truth and people put far more faith than they ought to in the results of fresh publications that have results they like (or fear).
gonational 6y ago
“reproducibility crisis” is a euphemism for “fake science”, which now, by the numbers, accounts for the majority of mainstream establishment science. Prove me wrong.
- fsh 6y ago
  "Fake science" implies intent to mislead. The reproducibility crisis is usually attributed to selection bias due to only positive results getting published.
  - gonational 6y ago
    You could have five times the accuracy by simply flipping a coin to answer any yes or no question, compared to the level of accuracy in the “reproducibility crisis”. I don’t think this can be explained by “selection bias”, and, if it can, and I do believe that could also qualify the entire industry as “fake science”, if the industry cannot even get close to as accurate as a coin toss.
    - xapata 6y ago
      If you'd read the literature on the reproducibility issue, you'd understand the selection mechanism can easily create the observed phenomenon. By calling it "fake science" you're touching on the idea of a paradigm shift, that periodically the community changes its beliefs about what enables someone to know something or to be certain about something.
      - gonational 6y ago
        My thinking is that, if he’s published papers are a large part of the means by which we share scientific knowledge, then, regardless of the underlying mechanism to be blamed (e.g., selection bias), the entire institution has become corrupted. To illustrate, the literature that you’re describing is presumably in the form of meta-analyses published in Nature. How can we even trust that to be “reproducible” (aka correct)? I think the correct reaction to this situation is to completely stop trusting scientists and their papers, in general. Not trusting scientists add their papers doesn’t mean disbelieving science in general; it just means the established institutions do not know how to do science correctly, and I believe this has been the case since the beginning of modern science. That means we can learn things by reading between the lines and using some common sense, but we can’t simply rely on institutions to tell us the correct answer to anything whatsoever.

umvi 6y ago

-- @mwilcox[0] [0] https://twitter.com/mwilcox/status/1260458110596120576 https://twitter.com/mwilcox/status/1260458110596120576

0xcafecafe 6y ago

Can someone please elaborate on this? What is the "reproducibility crisis" in this case?

xapata 6y ago
"Most published research findings are false" (not reproducible) https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1182327/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1182327/
- whydoyoucare 6y ago
  This is an excellent read: https://slatestarcodex.com/2014/12/12/beware-the-man-of-one-study/ https://slatestarcodex.com/2014/12/12/beware-the-man-of-one-... -- basically says one has to be very skeptical of trusting a study that suits the narrative, and how studies can be cherry-picked to align with your narrative.
umvi 6y ago
Reproducibility crisis is well known, especially in social sciences. However, it affects other science fields as well. In a nutshell, the crisis is that there is far more incentive to churn out a new publication/study than reproduce an existing one. So we have tons of studies that are the only one of its kind that have not been reproduced. The crisis part comes in where attempts to reproduce the results of these studies fail which throws into question the legitimacy of the entire field's publications. So you can imagine how putting your faith into covid-19 related studies (such as effects of hydroxychloroquine) that were rushed and results not reproduced independently can result in problems down the road. Science is probably the best tool we have for determining truth. However, it takes a long time and a lot of effort to converge on the truth and people put far more faith than they ought to in the results of fresh publications that have results they like (or fear).

gonational 6y ago

“reproducibility crisis” is a euphemism for “fake science”, which now, by the numbers, accounts for the majority of mainstream establishment science. Prove me wrong.

fsh 6y ago
"Fake science" implies intent to mislead. The reproducibility crisis is usually attributed to selection bias due to only positive results getting published.
- gonational 6y ago
  You could have five times the accuracy by simply flipping a coin to answer any yes or no question, compared to the level of accuracy in the “reproducibility crisis”. I don’t think this can be explained by “selection bias”, and, if it can, and I do believe that could also qualify the entire industry as “fake science”, if the industry cannot even get close to as accurate as a coin toss.
  - xapata 6y ago
    If you'd read the literature on the reproducibility issue, you'd understand the selection mechanism can easily create the observed phenomenon. By calling it "fake science" you're touching on the idea of a paradigm shift, that periodically the community changes its beliefs about what enables someone to know something or to be certain about something.
    - gonational 6y ago
      My thinking is that, if he’s published papers are a large part of the means by which we share scientific knowledge, then, regardless of the underlying mechanism to be blamed (e.g., selection bias), the entire institution has become corrupted. To illustrate, the literature that you’re describing is presumably in the form of meta-analyses published in Nature. How can we even trust that to be “reproducible” (aka correct)? I think the correct reaction to this situation is to completely stop trusting scientists and their papers, in general. Not trusting scientists add their papers doesn’t mean disbelieving science in general; it just means the established institutions do not know how to do science correctly, and I believe this has been the case since the beginning of modern science. That means we can learn things by reading between the lines and using some common sense, but we can’t simply rely on institutions to tell us the correct answer to anything whatsoever.