3 ms·

There is a serious ethical problem with submitting dummy papers in that it abuses the trust of the editors and reviewers. You are wasting the time of volunteer

There is a serious ethical problem with submitting dummy papers in that it abuses the trust of the editors and reviewers. You are wasting the time of volunteer reviewers who have nothing to gain but plenty to lose if they screw up. Reviewers assume papers are submitted in good faith, and are not traps. Editors have practice spotting cranks but trap papers don't have that smell about them since they are written by smart but naughty people trying to trick reviewers.

loup-vaillant 13y ago
If the author is to be believed, the flaws of the paper were numerous, obvious, and fatal. Meaning, easy to spot in five minutes by any reader with a bit of domain knowledge. Those are not subtle traps. Now, while I understand how journals can assume good faith, they should never assume competence. They are supposed to make sure the paper is good, dammit! If they let that one slip through, they are bound to let genuine fatal mistakes slip through. No matter what, papers should go through actual peer review before publication. This study, if conducted the way I have read it, is solid evidence that many journals do not do that. They broke trust in the first place.
betterunix 13y ago
If nobody ever tests the review process, how can we know that it is working?
- robotresearcher 13y ago
  We read the journals. Many readers of an article are qualified to review it. Many also personally know the editor or associate editors and will raise a stink if the journal starts to suck. We _are_ these journals. We write the articles for free, we review them for free, we proudly share our results. We hate it when a shitty paper gets in. Edit: There _are_ shitty journals that no one reads. Everyone in the business knows which ones these are. There is a problem explaining this to lay people, since the shitty journal web site doesn't have a sticker on it saying "zero credibility - please ignore these articles". This is not really a problem for researchers.
  - betterunix 13y ago
    That is not relevant to the issue of whether or not the peer review process works in deciding what articles to publish. Researchers might be able to identify bad journals and bad articles (though in my experience, quite a few researchers only spend time reading the results), but the point of the review process is to ensure that bad articles are not published in the first place. If the reviewers are allowing bad articles to slip through the cracks, then the process is not working and we need to seriously reconsider the entire journal system (though I would say that the Internet already requires us to reconsider that system). My real point was that submitting blatantly bad articles serves a purpose, which is to test whether the current process of reviewing articles is effective at weeding out bad articles. We should not blindly assume that journals, even top-tier journals, are only publishing good articles.
    - robotresearcher 13y ago
      I addressed this point in the third sentence: "[Researchers] will raise a stink if the journal starts to suck."
  - betterunix 13y ago
    That is not relevant to the issue of whether or not the peer review process works in deciding what articles to publish. Researchers might be able to identify bad journals and bad articles (though in my experience, quite a few researchers only spend time reading the results), but the point of the review process is to ensure that bad articles are not published in the first place. If the reviewers are allowing bad articles to slip through the cracks, then the process is not working and we need to seriously reconsider the entire journal system (though I would say that the Internet already requires us to reconsider that system). My real point was that submitting blatantly bad articles serves a purpose, which is to test whether the current process of reviewing articles is effective at weeding out bad articles. We should not blindly assume that journals, even top-tier journals, are only publishing good articles.
  - betterunix 13y ago
    That is not relevant to the issue of whether or not the peer review process works in deciding what articles to publish. Researchers might be able to identify bad journals and bad articles (though in my experience, quite a few researchers only spend time reading the results), but the point of the review process is to ensure that bad articles are not published in the first place. If the reviewers are allowing bad articles to slip through the cracks, then the process is not working and we need to seriously reconsider the entire journal system (though I would say that the Internet already requires us to reconsider that system). My real point was that submitting blatantly bad articles serves a purpose, which is to test whether the current process of reviewing articles is effective at weeding out bad articles. We should not blindly assume that journals, even top-tier journals, are only publishing good articles.

loup-vaillant 13y ago

If the author is to be believed, the flaws of the paper were numerous, obvious, and fatal. Meaning, easy to spot in five minutes by any reader with a bit of domain knowledge. Those are not subtle traps. Now, while I understand how journals can assume good faith, they should never assume competence. They are supposed to make sure the paper is good, dammit! If they let that one slip through, they are bound to let genuine fatal mistakes slip through. No matter what, papers should go through actual peer review before publication. This study, if conducted the way I have read it, is solid evidence that many journals do not do that. They broke trust in the first place.

betterunix 13y ago

If nobody ever tests the review process, how can we know that it is working?

robotresearcher 13y ago
We read the journals. Many readers of an article are qualified to review it. Many also personally know the editor or associate editors and will raise a stink if the journal starts to suck. We _are_ these journals. We write the articles for free, we review them for free, we proudly share our results. We hate it when a shitty paper gets in. Edit: There _are_ shitty journals that no one reads. Everyone in the business knows which ones these are. There is a problem explaining this to lay people, since the shitty journal web site doesn't have a sticker on it saying "zero credibility - please ignore these articles". This is not really a problem for researchers.
- betterunix 13y ago
  That is not relevant to the issue of whether or not the peer review process works in deciding what articles to publish. Researchers might be able to identify bad journals and bad articles (though in my experience, quite a few researchers only spend time reading the results), but the point of the review process is to ensure that bad articles are not published in the first place. If the reviewers are allowing bad articles to slip through the cracks, then the process is not working and we need to seriously reconsider the entire journal system (though I would say that the Internet already requires us to reconsider that system). My real point was that submitting blatantly bad articles serves a purpose, which is to test whether the current process of reviewing articles is effective at weeding out bad articles. We should not blindly assume that journals, even top-tier journals, are only publishing good articles.
  - robotresearcher 13y ago
    I addressed this point in the third sentence: "[Researchers] will raise a stink if the journal starts to suck."
- betterunix 13y ago
  That is not relevant to the issue of whether or not the peer review process works in deciding what articles to publish. Researchers might be able to identify bad journals and bad articles (though in my experience, quite a few researchers only spend time reading the results), but the point of the review process is to ensure that bad articles are not published in the first place. If the reviewers are allowing bad articles to slip through the cracks, then the process is not working and we need to seriously reconsider the entire journal system (though I would say that the Internet already requires us to reconsider that system). My real point was that submitting blatantly bad articles serves a purpose, which is to test whether the current process of reviewing articles is effective at weeding out bad articles. We should not blindly assume that journals, even top-tier journals, are only publishing good articles.
- betterunix 13y ago
  That is not relevant to the issue of whether or not the peer review process works in deciding what articles to publish. Researchers might be able to identify bad journals and bad articles (though in my experience, quite a few researchers only spend time reading the results), but the point of the review process is to ensure that bad articles are not published in the first place. If the reviewers are allowing bad articles to slip through the cracks, then the process is not working and we need to seriously reconsider the entire journal system (though I would say that the Internet already requires us to reconsider that system). My real point was that submitting blatantly bad articles serves a purpose, which is to test whether the current process of reviewing articles is effective at weeding out bad articles. We should not blindly assume that journals, even top-tier journals, are only publishing good articles.