6 ms·

This is a well known problem of scientific publications now-a-days: https://www.nature.com/news/1-500-scientists-lift-the-lid-on-reproducibility-1.19970 https:

by gnusci 8y ago

This is a well known problem of scientific publications now-a-days: https://www.nature.com/news/1-500-scientists-lift-the-lid-on-reproducibility-1.19970 https://www.nature.com/news/1-500-scientists-lift-the-lid-on... https://en.wikipedia.org/wiki/Replication_crisis https://en.wikipedia.org/wiki/Replication_crisis Papers have become a target for success, so scientist need publications for better status and remuneration: http://www.sciencemag.org/news/2017/08/cash-bonuses-peer-reviewed-papers-go-global http://www.sciencemag.org/news/2017/08/cash-bonuses-peer-rev... A new scientific "mafia" is in place around the world: https://www.technologyreview.com/s/608266/the-truth-about-chinas-cash-for-publication-policy/ https://www.technologyreview.com/s/608266/the-truth-about-ch... https://retractionwatch.com/2017/08/10/paid-publish-not-just-china/ https://retractionwatch.com/2017/08/10/paid-publish-not-just... So papers have become a good business, no the way to disseminate outstanding research results.

lsh 8y ago
> So papers have become a good business, no the way to disseminate outstanding research results. That's awfully cynical and over-broad, but I agree to a point. Greedy and unscrupulous publishers are part of the problem, but so are lax or unprincipled scientists eager for prestige and a career-making publication in a top tier journal. It's an unfortunate chicken-and-egg cycle now with no easy way to cut it. Perhaps more emphasis on replication post-publication? Perhaps a reputation system for unethical publishers or scientists?
- timr 8y ago
  "Greedy and unscrupulous publishers are part of the problem, but so are lax or unprincipled scientists eager for prestige and a career-making publication in a top tier journal." That's just incredibly unfair. There are some fields and methodologies where p-hacking and cherry-picking have been a problem, but the primary reason that papers aren't reproducible is just noise and basic statistics. As a scientist, you control for what you can think of, but there are often way too many variables to control completely, and it's probable that you miss some. Those variables come to light when someone else tries to work with your method and can't reproduce it locally. However, real scientists don't stop and accuse the original authors of being "unprincipled" -- nine-point-nine times out of ten, they work with the original authors to discover the discrepancies. It isn't surprising at all to actual, working scientists that most papers are impossible to reproduce from a clean room, using only the original paper. It's the expected state of affairs when you're working with imperfect, noisy techniques, and trying tease out subtle phenomena.
  - michaelt 8y ago
    There are some fields and methodologies where p-hacking and cherry-picking have been a problem, but the primary reason that papers aren't reproducible is just noise and basic statistics. It's possible to imagine a version of academia where results that can be attributed to noise don't get published.
    - wyattpeak 8y ago
      Is it? That doesn't seem at all obvious to me. In fact it seems decidedly impossible. Almost any result could in principle be attributable to noise; where are you planning to source all of the funding to run large enough studies to minimise that? And no matter how large your experiments or how many you run, you're still going to end up with some published results attributable to noise since, as GP says, that's the nature of statistics. By its nature, you cannot tell whether a result is noise. You only have odds. I'm not saying there aren't problems with reproducability in many fields, but to suggest that you can eliminate it entirely is naive. No, not naive - wrong.
      - TeMPOraL 8y ago
        > Almost any result could in principle be attributable to noise; where are you planning to source all of the funding to run large enough studies to minimise that? By its nature, you cannot tell whether a result is noise. You only have odds. Well, with a single paper the odds indeed are that it's noise. That's why we need reproduction. Now of course a paper needs to be published for it to be replicated later. But the paper (and/or supplemental material) should contain all possible things the research team can think of that are relevant to reproducing it - otherwise it's setting itself up to be unverifiable in practice. Papers that are unverifiable in practice should not be publishable at all, because a) they won't be reproduced and thus it'll be forever indistinguishable from noise, and b) there's no way to determine whether it's real research, or a cleverly crafted bullshit.
        wyattpeak 8y ago
        I don't disagree with any of that, although I'd stick a big citation needed on the implicit suggestion that there's a large group of scientists who aren't making a good-faith effort to ensure that their successors will have the information they need to reproduce (that is, after all, what a paper is). My issue is the flippant and silly claim that "[i]t's possible to imagine a version of academia where results that can be attributed to noise don't get published".
- gnusci 8y ago
  I totally agree that predatory publishers are a huge part of the problem: https://en.wikipedia.org/wiki/Predatory_open-access_publishing https://en.wikipedia.org/wiki/Predatory_open-access_publishi... https://jefferson.libguides.com/predatorypublishing https://jefferson.libguides.com/predatorypublishing https://beallslist.weebly.com/ https://beallslist.weebly.com/ And now predatory conferences: https://en.wikipedia.org/wiki/Predatory_conference https://en.wikipedia.org/wiki/Predatory_conference https://libguides.caltech.edu/c.php?g=512665&p=3503029 https://libguides.caltech.edu/c.php?g=512665&p=3503029 However, the storm actually started with the failure of reproducible of research in high impact journals: https://www.frontiersin.org/articles/10.3389/fnhum.2018.00037/full https://www.frontiersin.org/articles/10.3389/fnhum.2018.0003... http://www.sciencemag.org/news/2016/02/if-you-fail-reproduce-another-scientist-s-results-journal-wants-know http://www.sciencemag.org/news/2016/02/if-you-fail-reproduce... https://www.timeshighereducation.com/features/reproducing-results-how-big-is-the-problem https://www.timeshighereducation.com/features/reproducing-re... This is not an chicken-egg problem, it is an ethic problem, motivated by remuneration and funding.
  - lsh 8y ago
    Thankyou for the links
avip 8y ago
May I add http://slatestarcodex.com/2014/04/28/the-control-group-is-out-of-control/ http://slatestarcodex.com/2014/04/28/the-control-group-is-ou... to your list (was posted multiple times on HN)

by gnusci 8y ago

lsh 8y ago

> So papers have become a good business, no the way to disseminate outstanding research results. That's awfully cynical and over-broad, but I agree to a point. Greedy and unscrupulous publishers are part of the problem, but so are lax or unprincipled scientists eager for prestige and a career-making publication in a top tier journal. It's an unfortunate chicken-and-egg cycle now with no easy way to cut it. Perhaps more emphasis on replication post-publication? Perhaps a reputation system for unethical publishers or scientists?

timr 8y ago
"Greedy and unscrupulous publishers are part of the problem, but so are lax or unprincipled scientists eager for prestige and a career-making publication in a top tier journal." That's just incredibly unfair. There are some fields and methodologies where p-hacking and cherry-picking have been a problem, but the primary reason that papers aren't reproducible is just noise and basic statistics. As a scientist, you control for what you can think of, but there are often way too many variables to control completely, and it's probable that you miss some. Those variables come to light when someone else tries to work with your method and can't reproduce it locally. However, real scientists don't stop and accuse the original authors of being "unprincipled" -- nine-point-nine times out of ten, they work with the original authors to discover the discrepancies. It isn't surprising at all to actual, working scientists that most papers are impossible to reproduce from a clean room, using only the original paper. It's the expected state of affairs when you're working with imperfect, noisy techniques, and trying tease out subtle phenomena.
- michaelt 8y ago
  There are some fields and methodologies where p-hacking and cherry-picking have been a problem, but the primary reason that papers aren't reproducible is just noise and basic statistics. It's possible to imagine a version of academia where results that can be attributed to noise don't get published.
  - wyattpeak 8y ago
    Is it? That doesn't seem at all obvious to me. In fact it seems decidedly impossible. Almost any result could in principle be attributable to noise; where are you planning to source all of the funding to run large enough studies to minimise that? And no matter how large your experiments or how many you run, you're still going to end up with some published results attributable to noise since, as GP says, that's the nature of statistics. By its nature, you cannot tell whether a result is noise. You only have odds. I'm not saying there aren't problems with reproducability in many fields, but to suggest that you can eliminate it entirely is naive. No, not naive - wrong.
    - TeMPOraL 8y ago
      > Almost any result could in principle be attributable to noise; where are you planning to source all of the funding to run large enough studies to minimise that? By its nature, you cannot tell whether a result is noise. You only have odds. Well, with a single paper the odds indeed are that it's noise. That's why we need reproduction. Now of course a paper needs to be published for it to be replicated later. But the paper (and/or supplemental material) should contain all possible things the research team can think of that are relevant to reproducing it - otherwise it's setting itself up to be unverifiable in practice. Papers that are unverifiable in practice should not be publishable at all, because a) they won't be reproduced and thus it'll be forever indistinguishable from noise, and b) there's no way to determine whether it's real research, or a cleverly crafted bullshit.
      - wyattpeak 8y ago
        I don't disagree with any of that, although I'd stick a big citation needed on the implicit suggestion that there's a large group of scientists who aren't making a good-faith effort to ensure that their successors will have the information they need to reproduce (that is, after all, what a paper is). My issue is the flippant and silly claim that "[i]t's possible to imagine a version of academia where results that can be attributed to noise don't get published".
gnusci 8y ago
I totally agree that predatory publishers are a huge part of the problem: https://en.wikipedia.org/wiki/Predatory_open-access_publishing https://en.wikipedia.org/wiki/Predatory_open-access_publishi... https://jefferson.libguides.com/predatorypublishing https://jefferson.libguides.com/predatorypublishing https://beallslist.weebly.com/ https://beallslist.weebly.com/ And now predatory conferences: https://en.wikipedia.org/wiki/Predatory_conference https://en.wikipedia.org/wiki/Predatory_conference https://libguides.caltech.edu/c.php?g=512665&p=3503029 https://libguides.caltech.edu/c.php?g=512665&p=3503029 However, the storm actually started with the failure of reproducible of research in high impact journals: https://www.frontiersin.org/articles/10.3389/fnhum.2018.00037/full https://www.frontiersin.org/articles/10.3389/fnhum.2018.0003... http://www.sciencemag.org/news/2016/02/if-you-fail-reproduce-another-scientist-s-results-journal-wants-know http://www.sciencemag.org/news/2016/02/if-you-fail-reproduce... https://www.timeshighereducation.com/features/reproducing-results-how-big-is-the-problem https://www.timeshighereducation.com/features/reproducing-re... This is not an chicken-egg problem, it is an ethic problem, motivated by remuneration and funding.
- lsh 8y ago
  Thankyou for the links

avip 8y ago

May I add http://slatestarcodex.com/2014/04/28/the-control-group-is-out-of-control/ http://slatestarcodex.com/2014/04/28/the-control-group-is-ou... to your list (was posted multiple times on HN)