3 ms·

Suspicious phrases in peer reviews point to referees gaming the system

21 points by XzetaU8 2y ago

deleted 2y ago
[deleted]
derbOac 2y ago
I haven't had a chance to read the original paper but as an editor and frequent reviewer I had a couple of questions. First, a lot of journals post criteria to evaluate papers by. For various reasons, open journals seem to push these more strictly. Some of them get adopted across journals and even publishers. I could see a reviewer using the exact language of these criteria just to make it clear they're following the criteria. Second, reviewing is kind of in crisis, ironically for reasons that cut against the the argument of the paper. It's often thankless and increasingly hard to find reviewers. When you can they're short on time. These boilerplate bits of text or even use of AI might just be a way to save time while still acting in good faith. I don't mean to suggest there's nothing problematic going on or that the authors didn't address these issues but I can think of other interpretations of things that would be important to address.
readthenotes1 2y ago
And then there's the new rector of the University of Salamanca. https://english.elpais.com/science-tech/2024-05-31/internal-messages-show-how-the-new-head-of-one-of-the-worlds-oldest-universities-organized-a-citation-cartel.html https://english.elpais.com/science-tech/2024-05-31/internal-... Also see https://en.m.wikipedia.org/wiki/Goodhart's_law https://en.m.wikipedia.org/wiki/Goodhart's_law
gus_massa 2y ago
> “In abstract, the author should add more scientific findings.” “Discuss the novelty and clear application of the work in the abstract as well as in introduction section.” I don't understand why that are smoking guns. They look like standard problems that can be addressed with standard phrases. Do they appear in absolutely all the reviews of some reviewers? It's like “We recommend that the paper be accepted after revision.” The interesting part is if the paper makes sense and if the list of recommended revisions make sense, not that they copy&pasted a filler sentence.
hyperpape 2y ago
As a former academic (grad student, not professor), and now professional software developer (so exposed to open source), the idea of self-plagiarism irritates me a little bit, because of its vagueness. Of course, there is such a thing as "self-plagiarism". If you took an entire paper and resubmitted it to a different journal after it was published, that would be unethical. There is a related case that's fine where a book collects previously published papers, but that's done with full disclosure. However, I also think that large portions of what is published should be reused, and the norms surrounding that reuse are suboptimal. The background section for two papers often should have many sentences or paragraphs that are identical down to the word.[0] I certainly remember some fruitless sessions wondering whether a vocabulary change between two papers was load-bearing or not--encouragine reuse and making it explicit when components were reused would make papers more readable. In this particular case, I suspect that some of these reviewers were putting out shoddy reviews using copy-paste without thinking about it. But it's also possible some of them had just gotten good at copy-pasting the parts of reviews that are always boilerplate, and the non-repeated parts of their reviews are worthwhile. [0] The fact that you often skip those sections when reading scientific or mathematical papers is an argument for reuse, not against it. In philosophy, those sections often aren't really skippable, because terminology too often has small idiosyncracies that are introduced early on.

21 points by XzetaU8 2y ago

deleted 2y ago

[deleted]

I haven't had a chance to read the original paper but as an editor and frequent reviewer I had a couple of questions. First, a lot of journals post criteria to evaluate papers by. For various reasons, open journals seem to push these more strictly. Some of them get adopted across journals and even publishers. I could see a reviewer using the exact language of these criteria just to make it clear they're following the criteria. Second, reviewing is kind of in crisis, ironically for reasons that cut against the the argument of the paper. It's often thankless and increasingly hard to find reviewers. When you can they're short on time. These boilerplate bits of text or even use of AI might just be a way to save time while still acting in good faith. I don't mean to suggest there's nothing problematic going on or that the authors didn't address these issues but I can think of other interpretations of things that would be important to address.

readthenotes1 2y ago

And then there's the new rector of the University of Salamanca. https://english.elpais.com/science-tech/2024-05-31/internal-messages-show-how-the-new-head-of-one-of-the-worlds-oldest-universities-organized-a-citation-cartel.html https://english.elpais.com/science-tech/2024-05-31/internal-... Also see https://en.m.wikipedia.org/wiki/Goodhart's_law https://en.m.wikipedia.org/wiki/Goodhart's_law

gus_massa 2y ago

> “In abstract, the author should add more scientific findings.” “Discuss the novelty and clear application of the work in the abstract as well as in introduction section.” I don't understand why that are smoking guns. They look like standard problems that can be addressed with standard phrases. Do they appear in absolutely all the reviews of some reviewers? It's like “We recommend that the paper be accepted after revision.” The interesting part is if the paper makes sense and if the list of recommended revisions make sense, not that they copy&pasted a filler sentence.

hyperpape 2y ago

As a former academic (grad student, not professor), and now professional software developer (so exposed to open source), the idea of self-plagiarism irritates me a little bit, because of its vagueness. Of course, there is such a thing as "self-plagiarism". If you took an entire paper and resubmitted it to a different journal after it was published, that would be unethical. There is a related case that's fine where a book collects previously published papers, but that's done with full disclosure. However, I also think that large portions of what is published should be reused, and the norms surrounding that reuse are suboptimal. The background section for two papers often should have many sentences or paragraphs that are identical down to the word.[0] I certainly remember some fruitless sessions wondering whether a vocabulary change between two papers was load-bearing or not--encouragine reuse and making it explicit when components were reused would make papers more readable. In this particular case, I suspect that some of these reviewers were putting out shoddy reviews using copy-paste without thinking about it. But it's also possible some of them had just gotten good at copy-pasting the parts of reviews that are always boilerplate, and the non-repeated parts of their reviews are worthwhile. [0] The fact that you often skip those sections when reading scientific or mathematical papers is an argument for reuse, not against it. In philosophy, those sections often aren't really skippable, because terminology too often has small idiosyncracies that are introduced early on.