3 ms·

It would be even better to establish a global standard on peer review and adhere to it. A first step would be to move to a preprint model and require full disc

by sddfd 9y ago

It would be even better to establish a global standard on peer review and adhere to it. A first step would be to move to a preprint model and require full disclosure of data sets, turn in of source code, and pre-registration of clinical studies. In turn reviewers must be required to publish their reviews and sign them with their name. This would help keeping the business honest on both sides of peer review.

gboudrias 9y ago
Require? Who would be enforcing this standard? The law? As of now, journals decide who gets to be published, and the scientific community (or in an indirect sense, the public) decides which journals get read. So the scientific community has to come to a global consensus, or all countries must come to a consensus and enforce it legally (whereas we can't even agree that nukes are bad). Once all that's done, we still need trust. Researchers can remove data selectively, use altered versions of published code, etc. To say nothing of countries who would pretend to adopt the law without enforcing it, with or without malicious intent. We need trust, and trust is built on a case-by-case basis, it requires judgment. We as a community simply need to be more critical with new research.
- sddfd 9y ago
  I agree with your general argument. However, I don't think our first priority should be trust. Our first priority should be public availability of data and reviews. With this information, we get reproducibility and accountability for wrongdoing. In the long run, this will be worth more than trust.
- JumpCrisscross 9y ago
  > the scientific community has to come to a global consensus, or all countries must come to a consensus and enforce it legally The scientific community isn't homogenous. Some communities are better organized than others. This is visible in their varying quality and journals' openness (e.g. arXiv). The communities with strong nexuses with industry, e.g. oncology, would stand to enforce reproducibility the best. A grand bargain whereby pharmaceutical companies (and others who would commercially benefit from more-reproducible research), or even just the NIH, include reasonable standards as part of their grant requirements would move the needle forward. It would also help us troubleshoot what works and what doesn't.

by sddfd 9y ago

gboudrias 9y ago

Require? Who would be enforcing this standard? The law? As of now, journals decide who gets to be published, and the scientific community (or in an indirect sense, the public) decides which journals get read. So the scientific community has to come to a global consensus, or all countries must come to a consensus and enforce it legally (whereas we can't even agree that nukes are bad). Once all that's done, we still need trust. Researchers can remove data selectively, use altered versions of published code, etc. To say nothing of countries who would pretend to adopt the law without enforcing it, with or without malicious intent. We need trust, and trust is built on a case-by-case basis, it requires judgment. We as a community simply need to be more critical with new research.

sddfd 9y ago
I agree with your general argument. However, I don't think our first priority should be trust. Our first priority should be public availability of data and reviews. With this information, we get reproducibility and accountability for wrongdoing. In the long run, this will be worth more than trust.
JumpCrisscross 9y ago
> the scientific community has to come to a global consensus, or all countries must come to a consensus and enforce it legally The scientific community isn't homogenous. Some communities are better organized than others. This is visible in their varying quality and journals' openness (e.g. arXiv). The communities with strong nexuses with industry, e.g. oncology, would stand to enforce reproducibility the best. A grand bargain whereby pharmaceutical companies (and others who would commercially benefit from more-reproducible research), or even just the NIH, include reasonable standards as part of their grant requirements would move the needle forward. It would also help us troubleshoot what works and what doesn't.