3 ms·

An excellent demonstration of how easy it is to manipulate data to find significant results - it brings to light (again) the pervasive issue of data dredging in

by wjb3 2y ago

An excellent demonstration of how easy it is to manipulate data to find significant results - it brings to light (again) the pervasive issue of data dredging in research, where researchers, intentionally or unintentionally, keep testing hypotheses until they find something publishable, thereby undermining the integrity of scientific findings and highlighting the importance of preregistration and replication studies. How can we shift the academic incentives away from "publish or perish" toward promoting transparency and rigorous methodology in research? Are any of the current attempts moving the needle?

jarsdel 2y ago
Current attempts showing promise include initiatives like the Center for Open Science and journals such as eLife and PLOS ONE that emphasize methodological rigor over impact factor. Additionally, policies from organizations like the National Institutes of Health (NIH) mandating data sharing plans are gradually moving the needle toward these goals.
- nextos 2y ago
  eLife is very rigorous, but also very selective. Most submissions don't pass editorial consideration. I've found their reviews much more detailed and technical than Nature Medicine or Cell.
  - wjb3 2y ago
    When you say that eLife is "very rigorous," could you elaborate on what specific aspects of their review process or criteria you find particularly stringent, and how those compare to the review processes of Nature/Cell?
    - nextos 2y ago
      eLife publishes reviews very prominently within articles, and they ask all reviewers to discuss their assessment after they have evaluated the article. I think this favors deep reviews. In Nature or Cell, I have sometimes seen 2-liners sent to authors as a review, or reviews where the reviewer clearly read the article for just 10 minutes before rushing an opinion. I wish eLife doubled down on this and disclosed the reviewer names after e.g. a 2-year embargo. This would make reviews even better IMHO, more skin in the game.

by wjb3 2y ago

jarsdel 2y ago

Current attempts showing promise include initiatives like the Center for Open Science and journals such as eLife and PLOS ONE that emphasize methodological rigor over impact factor. Additionally, policies from organizations like the National Institutes of Health (NIH) mandating data sharing plans are gradually moving the needle toward these goals.

nextos 2y ago
eLife is very rigorous, but also very selective. Most submissions don't pass editorial consideration. I've found their reviews much more detailed and technical than Nature Medicine or Cell.
- wjb3 2y ago
  When you say that eLife is "very rigorous," could you elaborate on what specific aspects of their review process or criteria you find particularly stringent, and how those compare to the review processes of Nature/Cell?
  - nextos 2y ago
    eLife publishes reviews very prominently within articles, and they ask all reviewers to discuss their assessment after they have evaluated the article. I think this favors deep reviews. In Nature or Cell, I have sometimes seen 2-liners sent to authors as a review, or reviews where the reviewer clearly read the article for just 10 minutes before rushing an opinion. I wish eLife doubled down on this and disclosed the reviewer names after e.g. a 2-year embargo. This would make reviews even better IMHO, more skin in the game.