4 ms·

> several journals are pushing for reproducibility as a requirement The current requirement for academic publishing is peer-review, i.e.: "Colleagues in my fi

> several journals are pushing for reproducibility as a requirement The current requirement for academic publishing is peer-review, i.e.: "Colleagues in my field looked over it briefly and said it's ok [because it doesn't challenge their work]" This inevitably leads to whole fields of science being corrupted and biased in ways that we can't easily identify. Science publishing standards could be improved in any number of ways: e.g. original data must be shared in an open format, publish your hypotheses before doing the research, publish the code, standardise the setting out of methods and statistics and results so that other scientists in and out of the field have a hope of following them. Nobody in the industry is motivated to change this, and hides behind the romantic ideal of a scientist pushing the boundaries and breaking the rules to discover novel theories. If such a scientist exists today, we can't identify them against the background of noise.

alisonkisk 5y ago
Modern science doesn't have peer review! Peer review works within a community, scientists review each other's work to decide who to trust. But the public and funding agencies are not part of that review! So why to we believe or fund the results? Either we are not their peers and should not engage with them, or we should be reviewing their work as part of the publication process. We also have "peers" in criminal prosecution. Would you allow a criminal trial where only the scientist's colleagues were in the jury?
- bonoboTP 5y ago
  The problem is that laypeople or bureaucrats can't judge highly specialized technical contributions. It's already difficult to do if your subfield is a different niche.
  - teachingassist 5y ago
    > It's already difficult to do if your subfield is a different niche. It's not that difficult, at least in science and science-adjacent fields. The scientific method still follows the same principles. If you are used to reading papers, you can tell when someone has elided important detail, or not included the evidence for their claims - or, if someone [or a whole subfield] is using technical language to obfuscate their work.
    - bonoboTP 5y ago
      There are many levels of analysis to a paper. Yours is part of it but not the full picture. One other thing is the novelty aspect. To assess that, you need to know the current state of the field. Also the best practices, known methodological pitfalls, judge what is a valid chain of reasoning and what is too speculative etc. It's far from simple and I often see outsiders online criticizing largely irrelevant things while missing core issues. There's no shortcut to expertise in a field. You need first hand experience and knowledge of the state of the art. Otherwise, if papers have to talk to laypeople or generic scientists, we'll see even more smokes and mirrors. The idea of peer review is that a work should be challenged by those best equipped to challenge it.
      - teachingassist 5y ago
        A journal could follow a set of statistical standards that could be examined and approved by experts outside of its narrow field. I don't claim that paper reviews have to be or can be perfect - only that they could be held to a better standard than they are now. We see plenty of avenues for publishing to a worse standard (self-publishing on an internet aggregator, for example). Let's see some avenues for a better standard. Why isn't this a science in itself? We could trial some different standards and statistically compare them. [It was not my claim that scientific papers must be understandable by laypeople, although I think this is an ideal case. If not, it should at least be understandable by people in an adjacent subfield - otherwise, it's just a small group of people talking amongst themselves]
    - Blikkentrekker 5y ago
      > The scientific method still follows the same principles. “the scientific method” is rather loosely defined and wide. What passes as “the scientific method” in some fields would not in others.
alisonkisk 5y ago
Modern science doesn't'have peer review! Peer review works within a community, scientists review each other's work to decide who to trust. But the public and funding agencies are not part of that review! Not even scientists in tangentially related fields (such as statisticians or methodology experts) are consulted. Furthermore, "peer review" is limited to pre publication. Follow up research by others has mininal effect on the impact of a paper on a scientist's career. So why to we believe or fund the results? Either we are not their peers and should not engage with them, or we should be reviewing their work as part of the publication process. We also have "peers" in criminal prosecution. Would you allow a criminal trial where only the scientist's colleagues were in the jury?
Blikkentrekker 5y ago
The reason it exists, is allowed to exist, and is no problem that it exists is that most science is infotainment, in that no one actually has any vested interest in it's accuracy. It's meant to be interesting to read, but for most science no one is banking on it's accuracy to the point that they would surely notice if it weren't accurate because, say, a plane would crash or a train would derail or a rocket would explode. Such does not happen with scientific research that forms the basis of the engineering on which many modern day applications are built. If the principles be wrong there, companies would loose substantial investment, so they are quite vested in verifying that it's accurate. Most science is simply infotainment for other scientists: it exists for no other reason than to be entertaining and fascinating to read, and might as well be fiction.

alisonkisk 5y ago

Modern science doesn't have peer review! Peer review works within a community, scientists review each other's work to decide who to trust. But the public and funding agencies are not part of that review! So why to we believe or fund the results? Either we are not their peers and should not engage with them, or we should be reviewing their work as part of the publication process. We also have "peers" in criminal prosecution. Would you allow a criminal trial where only the scientist's colleagues were in the jury?

bonoboTP 5y ago
The problem is that laypeople or bureaucrats can't judge highly specialized technical contributions. It's already difficult to do if your subfield is a different niche.
- teachingassist 5y ago
  > It's already difficult to do if your subfield is a different niche. It's not that difficult, at least in science and science-adjacent fields. The scientific method still follows the same principles. If you are used to reading papers, you can tell when someone has elided important detail, or not included the evidence for their claims - or, if someone [or a whole subfield] is using technical language to obfuscate their work.
  - bonoboTP 5y ago
    There are many levels of analysis to a paper. Yours is part of it but not the full picture. One other thing is the novelty aspect. To assess that, you need to know the current state of the field. Also the best practices, known methodological pitfalls, judge what is a valid chain of reasoning and what is too speculative etc. It's far from simple and I often see outsiders online criticizing largely irrelevant things while missing core issues. There's no shortcut to expertise in a field. You need first hand experience and knowledge of the state of the art. Otherwise, if papers have to talk to laypeople or generic scientists, we'll see even more smokes and mirrors. The idea of peer review is that a work should be challenged by those best equipped to challenge it.
    - teachingassist 5y ago
      A journal could follow a set of statistical standards that could be examined and approved by experts outside of its narrow field. I don't claim that paper reviews have to be or can be perfect - only that they could be held to a better standard than they are now. We see plenty of avenues for publishing to a worse standard (self-publishing on an internet aggregator, for example). Let's see some avenues for a better standard. Why isn't this a science in itself? We could trial some different standards and statistically compare them. [It was not my claim that scientific papers must be understandable by laypeople, although I think this is an ideal case. If not, it should at least be understandable by people in an adjacent subfield - otherwise, it's just a small group of people talking amongst themselves]
  - Blikkentrekker 5y ago
    > The scientific method still follows the same principles. “the scientific method” is rather loosely defined and wide. What passes as “the scientific method” in some fields would not in others.

alisonkisk 5y ago

Modern science doesn't'have peer review! Peer review works within a community, scientists review each other's work to decide who to trust. But the public and funding agencies are not part of that review! Not even scientists in tangentially related fields (such as statisticians or methodology experts) are consulted. Furthermore, "peer review" is limited to pre publication. Follow up research by others has mininal effect on the impact of a paper on a scientist's career. So why to we believe or fund the results? Either we are not their peers and should not engage with them, or we should be reviewing their work as part of the publication process. We also have "peers" in criminal prosecution. Would you allow a criminal trial where only the scientist's colleagues were in the jury?

Blikkentrekker 5y ago

The reason it exists, is allowed to exist, and is no problem that it exists is that most science is infotainment, in that no one actually has any vested interest in it's accuracy. It's meant to be interesting to read, but for most science no one is banking on it's accuracy to the point that they would surely notice if it weren't accurate because, say, a plane would crash or a train would derail or a rocket would explode. Such does not happen with scientific research that forms the basis of the engineering on which many modern day applications are built. If the principles be wrong there, companies would loose substantial investment, so they are quite vested in verifying that it's accurate. Most science is simply infotainment for other scientists: it exists for no other reason than to be entertaining and fascinating to read, and might as well be fiction.