9 ms·

Is Peer Review a Good Idea? (2020)

92 points by johntfella 4y ago

Stevenjobss 4y ago
jerojero 4y ago
I think ML, at least when I was involved in it, had a much more sane way of approaching publication than other fields (I came from a biology background). Open science addresses a lot of the pain points that are mentioned in the article. I think we can all agree that having certain magazines that include what can be considered the best works is a good thing; people for better or worse need these instruments to seamlessly judge the base quality of an article. Regardless of my personal criticisms and potential declining quality of a particular magazine it is obviously very helpful to be able to know "hey, at least I can expect some degree of quality given that it was published in this or another magazine" specially as a student. But this does not mean we ought to gate-keep research. Not only that, but having your publication out can open up a lot of feedback that can be incorporated and addressed. And this is where I go back to my original statement, in ML, it was very common to publish your work to arxiv for people to read through it and I think that greatly improved the speed upon which the field developed. Access is very important. So I'm all for abolishing "pre-publication peer review".
- hackernewds 4y ago
  This vastly underestimates the esoteric nature, experiential knowledge and complexity required to review most research published in the scientific community. The common public cannot be expected to provide non-populist capable review of pharmaceutical level research
  - jerojero 4y ago
    But they can do so for deep learning and mathematics? Because in those fields, as I said, pre-publication peer review is not as common. Of course we need peer-review and neither me nor the paper in the OP say otherwise. We just can't be gate-keeping research because of the review process; the review process needs to be re-thought and that includes the when and why.
Qem 4y ago
It would be a much better idea if double-blind peer review were a default. See "Nobel and Novice: Author Prominence Affects Peer Review": https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4190976 https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4190976
- RcouF1uZ4gsC 4y ago
  Double-blind peer review sounds nice in theory, but I don't think it would hold up in practice, especially in specialized domains. In those domains, everyone knows everyone else. Even if you hid the name, looking at the writing style, the subject, the choice of analysis software, the type of study. etc would largely give away who the authors are. For example I bet you could figure out that Richard Stallman wrote a particular paper, even if I removed his name from it. Or on HN, how many times have you read a comment and gone, "that sounds like..." even without fully noticing who posted it.
  - gernb 4y ago
    ...
    - mturmon 4y ago
      You seem to be talking about a different kind of peer review.
  - jiggawatts 4y ago
    Worse still, many papers are basically unintelligible to anyone outside of an "inner circle". Every time this is raised, inevitably people will pipe up and say "Oh, you're just not a <specialist in field>, that's why you can't read the paper!" For example, a new particle discovery paper was being debated in a forum and several people raised the issue that the paper was impenetrable, and was followed with the usual retorts of "you're not a particle physicist!". There was also a response from a particle physicist who works at CERN. He basically said that he himself could not understand the paper, despite working on a similar experiment on a different instrument. His office was down the hall from the office where the paper was written, but each team has their own terminology, jargon, and instrument-specific calibrations. Essentially, they can review their own papers, but nobody else can meaningfully validate it, or even understand it in a useful way! The problem with how modern science is performed is that it rewards the act of publishing a paper, not the utility, validity, or comprehensibility of the paper. In fact, writing papers to superficially look more 'serious' is rewarded, so papers that start off quite readable are often edited to be more obtuse and full of unnecessary symbols, terminology, or equations. I noticed this in Computer Science, where earlier papers in the field from the 60s and 70s were very readable, but then slowly started getting filled with Greek symbols and obscure terminology for no reason. There is no long history of Ancient Greek Computer Science that needs to be preserved for continuity with papers published hundreds of years ago! This is pure intellectual wankery, and has resulted in people like me unable to follow some modern CS papers on topics that I'm already familiar with. PS: Imagine the quality of "open source libraries" if everyone got accolades for publishing their code to GitHub, NuGet, or NPM, but no reward for anything else. No in-place updates. No direct usage. No pull requests. No feedback other than a one-time rubber stamp. No issue tracker. Nothing. Just... publish your code once and walk away, you get a cookie. Doesn't even matter if it compiles. Doesn't matter if it's pseudo-code you made up yourself. Just get on there, and you get something. But you get nothing for anything else. Picture that. That's modern science.
sn41 4y ago
I've been in academic research for 22 years. Based on my experience, I'll go out on a limb and say that even single-blind reviews are to be discarded. This is leading to quite irresponsible reviews. Instead, authored and credited reviews might lead to more responsible reviews, or reviewers respectfully declining when they might not know the topic. Instead, in CS, there is a tendency to hide behind an abrasive negative review when the reality is that the reviewer does not understand the paper. Programme Committees are relieved to find a negative review, however unfair or off-kilter it is, because more rejected papers will decrease the acceptance ratio of the conference, hence make it appear more competitive. Double-blind reviews are just peer-review theater. It is quite simple to guess which group the paper is from. It is difficult to guess the exact set of authors, but reviewers who are out to settle a score or to discard dismissively just need to know a ballpark of where the paper is from in order to stonewall with an irascible review.
- Gimpei 4y ago
  The one experience I had with peer review before leaving academia was actually pretty nice. One of the reviewers was a bit of a hardass, but they made good points and I think made the paper much better. As a consumer of papers now, I find peer review pretty useful. I can’t read everything that comes out. Even if I just read abstracts, I need some sort of filter. Publication in a top tier journal raises the probability that egregious errors haven’t been made and that the paper is worthy of my time.
  - sn41 4y ago
    Peer review is useful, no doubt. It is only reviewer anonymity that probably should be re-examined.
    - bumby 4y ago
      There would probably be a need for a mechanism to ensure objectivity. E.g., if Reviewer A rejects Reviewer B, there may be a higher chance Reviewer B will reject Review A's future work, just out of spite.
- 4y ago
kentlyons 4y ago
While knowing who the authors are can cause bias, hiding behind anonymization (as a reviewer) can as well. I think publishing the reviews (and who reviews) would help a lot (some fields do this). Science is really a dialog. The authors are putting forth part of the dialog and the reviewers are reflecting on it and giving their opinions. Hopefully publishing names next to reviews would minimize crappy reviews (often by the most senior folks). However the other problem is there are not enough qualified reviewers to go around and I could see many opting out of putting their name next to a review. But, we're likely better off. There also often isn't a clear cut line between "good" and "bad". Almost all papers have flaws and putting those out in the open for others to improve upon, or at least acknowledge, would help move humanities knowledge forward.
dahfizz 4y ago
> I have a partial solution: researchers “publish” papers to arXiv or similar, then “submit” them to the journal, which conducts peer review. The “journal” is a list of links to papers that it has accepted or verified. This made me curious about how arxiv operates. It seems that you require endorsement to become a registered author, and the submissions are moderated[1][2]. This already seems sufficient to keep out spam and clearly junk science. What is the value add of official(?) journals? [1] https://arxiv.org/about https://arxiv.org/about [2] https://arxiv.org/help/submit https://arxiv.org/help/submit
- remram 4y ago
  Where is that quote from?
  - texaslonghorn5 4y ago
    https://jakeseliger.com/2020/05/24/a-simple-solution-to-peer-review-problems/ https://jakeseliger.com/2020/05/24/a-simple-solution-to-peer...
    - remram 4y ago
      Did you mean to reply to https://news.ycombinator.com/item?id=32832212 https://news.ycombinator.com/item?id=32832212 ?
      - texaslonghorn5 4y ago
        No? Maybe I misinterpreted your comment. I was answering your question > Where is that quote from? by providing a link to the source of the quote. Ironically, maybe the person to whom you first replied was actually intending to respond to the comment you just linked instead of making a top-level comment.
        remram 4y ago
        Not ironic, that is what I meant. I didn't realize they were two different people, my bad.
rgrieselhuber 4y ago
The standard should be reproducibility, not peer-reviewed. Reproducibility is open loop, peer-review only is very susceptible to closed loop.
- snowwrestler 4y ago
  Different categories. Papers are what peers review. Experiments are what are reproducible.
- bluGill 4y ago
  The standard isn't peer review, it is unusual. https://xkcd.com/882/ https://xkcd.com/882/ applies to science and will get you publishes that green jelly beans cause cancer with no need to mention all experiments that got expected results. Which is why reproducibility should be the most important topic.
- advisedwang 4y ago
  There's lots of science that can't be reproduced before publishing or where reproducing doesn't help: * Theoretical work * Computer simulations (re-run the simulation? Well that's not going to detect most issues. Re-create the program from scratch? expensive and hard to make a re-write meaningfully different systematically) * Cosmological observations (can't re-run that supernova!) * Medical case studies (can't just find a new patient) * Large scale studies (build a second LHC? Do results based on the Framingham Study have to kick off another 30 years of data collection before they publish) In fact it's only small and self contained experiments where this is really practical.
  - Mountain_Skies 4y ago
    >Computer simulations (re-run the simulation? Well that's not going to detect most issues It would have detected the fraud going on at ICL with their covid model that was used by the governments of the world to set the course of pandemic response. I can't think of better example of a situation where requiring replication would have altered the path of humanity as much as it would have here. Running the same simulation with the same inputs produced wildly different results between runs. That would have been an easily detected red flag but because no one was able to do so until long after the pandemic response was set in motion and changes would have been politically uncomfortable for the powers that be, we ended with a terrible response that many still cannot bring themselves to admit was both poor and instigated by a terrible model that should have been easily detected as being garbage.
ck2 4y ago
One of the best uses of Reddit is to find a thread where someone well educated is taking apart an academic paper. You will be amazed what you didn't spot/assumed and learn a lot.
- EEBio 4y ago
  Do you maybe have an example or two at hand? I would be interested to read some.
luispauloml 4y ago
Off-topic comment: it surprised me that The British Journal for the Philosophy of Science is published by The University of Chicago Press.
meowtastic 4y ago
Doesn't the proposed solution already exist? Authors are allowed to publish their papers in preprint repositories before submitting to a journal.
betwixthewires 4y ago
I like the basic principle of peer review, but not how it's done. I understand why it was done that way for so long, but we have the technology to do better now. I read an interview with a TV show runner who said you can come up with any crazy plot twist thing, and within 10 minutes of the first episode airing some guy on the internet has already figured the whole thing out. I think this phenomenon could be put to good use. Just publish your papers publicly with a comments section. If there's problems with it people will tear it apart. Source their work and let the world help you improve your work.
scientism 4y ago
The way Journal of Open Source (JOSS) - https://joss.theoj.org https://joss.theoj.org - does the reviewing process is good. Everything is out in the open, reviewers and authors names, reviews, responses, discussions etc. The whole process is out there for everyone to see.
marsa 4y ago
having worked in academic publishing, i've concluded that peer review would perform best if it were embedded in the very act of reading (so post-publication, non-formalized kind of peer review) instead of having it be a formalized process riddled with issues and cracks as it is currently. it would enable faster and more seamless communication, academics wouldn't be burdened with extra volunteer work, and publishers would still be able to curate works as the authors suggest (+ there are other ways to do it). sadly my impression is that it's simply too entrenched in the publishing process -- and publisher's raison d'etre to a large extent -- for the publishers to relinquish control of it
sylware 4y ago
The main issue with critical and complex code is that most if not all of your peers who are going to review your code are actually hackers..........
probably_wrong 4y ago
As someone who is inside the academic publication loop, I am in favor of double-blind, non-public peer reviews because I don't want to invite 4Chan into my daily work. I believe that double-blind, non-public reviews do a great job at protecting those in a more precarious situation: grad students are not judged by their lack of publication record, there is no permanent public record of all those times they got rejected, and reviewers can be both more honest and more certain that a rejection won't lead to a 4Chan/Twitter mob coming for them. According to the ACL's 2019 survey [1], "female respondents were less likely to support public review than male respondents". I'd be weary of implementing any changed that would make academia even more hostile to women and minorities. [1] http://acl2019pcblog.fileli.unipi.it/wp-content/uploads/2019/07/ReportACL2019ReviewingSurvey.pdf http://acl2019pcblog.fileli.unipi.it/wp-content/uploads/2019...
- LudwigNagasena 4y ago
  Interesting to see how the good ol’ “think of the children” defense has morphed into “think of the women and minorities”.
- jhbadger 4y ago
  I just don't see how double-blind peer review can work. To properly peer review a manuscript you often have to read the papers cited (because often details of the methods of the current manuscript are described in previously published work). These citations are almost always to the authors' own work. So it is trivial for any peer reviewer to figure out who wrote the manuscript they are reviewing, even if their names are blinded.
  - humanistbot 4y ago
    I hear this a lot and I just don't buy it. The whole idea of science is that you can describe methods in text, send that text around the world, and have other people use those same methods to do their own independent studies. So even if you know that lab X really likes using a particular method, it could be someone from lab Y who read the papers from lab X and wanted to use them as well. Sure, people like citing themselves. Editors of double-blind journals already are supposed to desk reject work that says something like "in our prior study (Smith 2020)..." They should also desk reject obnoxiously self-referential citations --- like "Great advances have been made in method X (Smith 2016, 2017a, 2017b, 2018, 2019a, 2019b, 2020, 2021)"
micheles 4y ago
Having been on both sides of the peer review process in my field (Theoretical Physics), I say we should completely abolish it. Just publish the papers on the preprint archives. Some will be bad, some will be good, and you will know the impact of a paper after 10 years, nothing different from how it is now. Currently the very big majority of the effort in the reviews (on both sides of the process) is wasted time.

92 points by johntfella 4y ago

I think ML, at least when I was involved in it, had a much more sane way of approaching publication than other fields (I came from a biology background). Open science addresses a lot of the pain points that are mentioned in the article. I think we can all agree that having certain magazines that include what can be considered the best works is a good thing; people for better or worse need these instruments to seamlessly judge the base quality of an article. Regardless of my personal criticisms and potential declining quality of a particular magazine it is obviously very helpful to be able to know "hey, at least I can expect some degree of quality given that it was published in this or another magazine" specially as a student. But this does not mean we ought to gate-keep research. Not only that, but having your publication out can open up a lot of feedback that can be incorporated and addressed. And this is where I go back to my original statement, in ML, it was very common to publish your work to arxiv for people to read through it and I think that greatly improved the speed upon which the field developed. Access is very important. So I'm all for abolishing "pre-publication peer review".

hackernewds 4y ago
This vastly underestimates the esoteric nature, experiential knowledge and complexity required to review most research published in the scientific community. The common public cannot be expected to provide non-populist capable review of pharmaceutical level research
- jerojero 4y ago
  But they can do so for deep learning and mathematics? Because in those fields, as I said, pre-publication peer review is not as common. Of course we need peer-review and neither me nor the paper in the OP say otherwise. We just can't be gate-keeping research because of the review process; the review process needs to be re-thought and that includes the when and why.

Qem 4y ago

It would be a much better idea if double-blind peer review were a default. See "Nobel and Novice: Author Prominence Affects Peer Review": https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4190976 https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4190976

RcouF1uZ4gsC 4y ago
Double-blind peer review sounds nice in theory, but I don't think it would hold up in practice, especially in specialized domains. In those domains, everyone knows everyone else. Even if you hid the name, looking at the writing style, the subject, the choice of analysis software, the type of study. etc would largely give away who the authors are. For example I bet you could figure out that Richard Stallman wrote a particular paper, even if I removed his name from it. Or on HN, how many times have you read a comment and gone, "that sounds like..." even without fully noticing who posted it.
- gernb 4y ago
  ...
  - mturmon 4y ago
    You seem to be talking about a different kind of peer review.
- jiggawatts 4y ago
  Worse still, many papers are basically unintelligible to anyone outside of an "inner circle". Every time this is raised, inevitably people will pipe up and say "Oh, you're just not a <specialist in field>, that's why you can't read the paper!" For example, a new particle discovery paper was being debated in a forum and several people raised the issue that the paper was impenetrable, and was followed with the usual retorts of "you're not a particle physicist!". There was also a response from a particle physicist who works at CERN. He basically said that he himself could not understand the paper, despite working on a similar experiment on a different instrument. His office was down the hall from the office where the paper was written, but each team has their own terminology, jargon, and instrument-specific calibrations. Essentially, they can review their own papers, but nobody else can meaningfully validate it, or even understand it in a useful way! The problem with how modern science is performed is that it rewards the act of publishing a paper, not the utility, validity, or comprehensibility of the paper. In fact, writing papers to superficially look more 'serious' is rewarded, so papers that start off quite readable are often edited to be more obtuse and full of unnecessary symbols, terminology, or equations. I noticed this in Computer Science, where earlier papers in the field from the 60s and 70s were very readable, but then slowly started getting filled with Greek symbols and obscure terminology for no reason. There is no long history of Ancient Greek Computer Science that needs to be preserved for continuity with papers published hundreds of years ago! This is pure intellectual wankery, and has resulted in people like me unable to follow some modern CS papers on topics that I'm already familiar with. PS: Imagine the quality of "open source libraries" if everyone got accolades for publishing their code to GitHub, NuGet, or NPM, but no reward for anything else. No in-place updates. No direct usage. No pull requests. No feedback other than a one-time rubber stamp. No issue tracker. Nothing. Just... publish your code once and walk away, you get a cookie. Doesn't even matter if it compiles. Doesn't matter if it's pseudo-code you made up yourself. Just get on there, and you get something. But you get nothing for anything else. Picture that. That's modern science.

sn41 4y ago

I've been in academic research for 22 years. Based on my experience, I'll go out on a limb and say that even single-blind reviews are to be discarded. This is leading to quite irresponsible reviews. Instead, authored and credited reviews might lead to more responsible reviews, or reviewers respectfully declining when they might not know the topic. Instead, in CS, there is a tendency to hide behind an abrasive negative review when the reality is that the reviewer does not understand the paper. Programme Committees are relieved to find a negative review, however unfair or off-kilter it is, because more rejected papers will decrease the acceptance ratio of the conference, hence make it appear more competitive. Double-blind reviews are just peer-review theater. It is quite simple to guess which group the paper is from. It is difficult to guess the exact set of authors, but reviewers who are out to settle a score or to discard dismissively just need to know a ballpark of where the paper is from in order to stonewall with an irascible review.

Gimpei 4y ago
The one experience I had with peer review before leaving academia was actually pretty nice. One of the reviewers was a bit of a hardass, but they made good points and I think made the paper much better. As a consumer of papers now, I find peer review pretty useful. I can’t read everything that comes out. Even if I just read abstracts, I need some sort of filter. Publication in a top tier journal raises the probability that egregious errors haven’t been made and that the paper is worthy of my time.
- sn41 4y ago
  Peer review is useful, no doubt. It is only reviewer anonymity that probably should be re-examined.
  - bumby 4y ago
    There would probably be a need for a mechanism to ensure objectivity. E.g., if Reviewer A rejects Reviewer B, there may be a higher chance Reviewer B will reject Review A's future work, just out of spite.
4y ago

kentlyons 4y ago

While knowing who the authors are can cause bias, hiding behind anonymization (as a reviewer) can as well. I think publishing the reviews (and who reviews) would help a lot (some fields do this). Science is really a dialog. The authors are putting forth part of the dialog and the reviewers are reflecting on it and giving their opinions. Hopefully publishing names next to reviews would minimize crappy reviews (often by the most senior folks). However the other problem is there are not enough qualified reviewers to go around and I could see many opting out of putting their name next to a review. But, we're likely better off. There also often isn't a clear cut line between "good" and "bad". Almost all papers have flaws and putting those out in the open for others to improve upon, or at least acknowledge, would help move humanities knowledge forward.

dahfizz 4y ago

> I have a partial solution: researchers “publish” papers to arXiv or similar, then “submit” them to the journal, which conducts peer review. The “journal” is a list of links to papers that it has accepted or verified. This made me curious about how arxiv operates. It seems that you require endorsement to become a registered author, and the submissions are moderated[1][2]. This already seems sufficient to keep out spam and clearly junk science. What is the value add of official(?) journals? [1] https://arxiv.org/about https://arxiv.org/about [2] https://arxiv.org/help/submit https://arxiv.org/help/submit

remram 4y ago
Where is that quote from?
- texaslonghorn5 4y ago
  https://jakeseliger.com/2020/05/24/a-simple-solution-to-peer-review-problems/ https://jakeseliger.com/2020/05/24/a-simple-solution-to-peer...
  - remram 4y ago
    Did you mean to reply to https://news.ycombinator.com/item?id=32832212 https://news.ycombinator.com/item?id=32832212 ?
    - texaslonghorn5 4y ago
      No? Maybe I misinterpreted your comment. I was answering your question > Where is that quote from? by providing a link to the source of the quote. Ironically, maybe the person to whom you first replied was actually intending to respond to the comment you just linked instead of making a top-level comment.
      - remram 4y ago
        Not ironic, that is what I meant. I didn't realize they were two different people, my bad.

rgrieselhuber 4y ago

The standard should be reproducibility, not peer-reviewed. Reproducibility is open loop, peer-review only is very susceptible to closed loop.

snowwrestler 4y ago
Different categories. Papers are what peers review. Experiments are what are reproducible.
bluGill 4y ago
The standard isn't peer review, it is unusual. https://xkcd.com/882/ https://xkcd.com/882/ applies to science and will get you publishes that green jelly beans cause cancer with no need to mention all experiments that got expected results. Which is why reproducibility should be the most important topic.
advisedwang 4y ago
There's lots of science that can't be reproduced before publishing or where reproducing doesn't help: * Theoretical work * Computer simulations (re-run the simulation? Well that's not going to detect most issues. Re-create the program from scratch? expensive and hard to make a re-write meaningfully different systematically) * Cosmological observations (can't re-run that supernova!) * Medical case studies (can't just find a new patient) * Large scale studies (build a second LHC? Do results based on the Framingham Study have to kick off another 30 years of data collection before they publish) In fact it's only small and self contained experiments where this is really practical.
- Mountain_Skies 4y ago
  >Computer simulations (re-run the simulation? Well that's not going to detect most issues It would have detected the fraud going on at ICL with their covid model that was used by the governments of the world to set the course of pandemic response. I can't think of better example of a situation where requiring replication would have altered the path of humanity as much as it would have here. Running the same simulation with the same inputs produced wildly different results between runs. That would have been an easily detected red flag but because no one was able to do so until long after the pandemic response was set in motion and changes would have been politically uncomfortable for the powers that be, we ended with a terrible response that many still cannot bring themselves to admit was both poor and instigated by a terrible model that should have been easily detected as being garbage.

ck2 4y ago

One of the best uses of Reddit is to find a thread where someone well educated is taking apart an academic paper. You will be amazed what you didn't spot/assumed and learn a lot.

EEBio 4y ago
Do you maybe have an example or two at hand? I would be interested to read some.

luispauloml 4y ago

Off-topic comment: it surprised me that The British Journal for the Philosophy of Science is published by The University of Chicago Press.

meowtastic 4y ago

Doesn't the proposed solution already exist? Authors are allowed to publish their papers in preprint repositories before submitting to a journal.

betwixthewires 4y ago

I like the basic principle of peer review, but not how it's done. I understand why it was done that way for so long, but we have the technology to do better now. I read an interview with a TV show runner who said you can come up with any crazy plot twist thing, and within 10 minutes of the first episode airing some guy on the internet has already figured the whole thing out. I think this phenomenon could be put to good use. Just publish your papers publicly with a comments section. If there's problems with it people will tear it apart. Source their work and let the world help you improve your work.

scientism 4y ago

The way Journal of Open Source (JOSS) - https://joss.theoj.org https://joss.theoj.org - does the reviewing process is good. Everything is out in the open, reviewers and authors names, reviews, responses, discussions etc. The whole process is out there for everyone to see.

marsa 4y ago

having worked in academic publishing, i've concluded that peer review would perform best if it were embedded in the very act of reading (so post-publication, non-formalized kind of peer review) instead of having it be a formalized process riddled with issues and cracks as it is currently. it would enable faster and more seamless communication, academics wouldn't be burdened with extra volunteer work, and publishers would still be able to curate works as the authors suggest (+ there are other ways to do it). sadly my impression is that it's simply too entrenched in the publishing process -- and publisher's raison d'etre to a large extent -- for the publishers to relinquish control of it

sylware 4y ago

The main issue with critical and complex code is that most if not all of your peers who are going to review your code are actually hackers..........

probably_wrong 4y ago

As someone who is inside the academic publication loop, I am in favor of double-blind, non-public peer reviews because I don't want to invite 4Chan into my daily work. I believe that double-blind, non-public reviews do a great job at protecting those in a more precarious situation: grad students are not judged by their lack of publication record, there is no permanent public record of all those times they got rejected, and reviewers can be both more honest and more certain that a rejection won't lead to a 4Chan/Twitter mob coming for them. According to the ACL's 2019 survey [1], "female respondents were less likely to support public review than male respondents". I'd be weary of implementing any changed that would make academia even more hostile to women and minorities. [1] http://acl2019pcblog.fileli.unipi.it/wp-content/uploads/2019/07/ReportACL2019ReviewingSurvey.pdf http://acl2019pcblog.fileli.unipi.it/wp-content/uploads/2019...

LudwigNagasena 4y ago
Interesting to see how the good ol’ “think of the children” defense has morphed into “think of the women and minorities”.
jhbadger 4y ago
I just don't see how double-blind peer review can work. To properly peer review a manuscript you often have to read the papers cited (because often details of the methods of the current manuscript are described in previously published work). These citations are almost always to the authors' own work. So it is trivial for any peer reviewer to figure out who wrote the manuscript they are reviewing, even if their names are blinded.
- humanistbot 4y ago
  I hear this a lot and I just don't buy it. The whole idea of science is that you can describe methods in text, send that text around the world, and have other people use those same methods to do their own independent studies. So even if you know that lab X really likes using a particular method, it could be someone from lab Y who read the papers from lab X and wanted to use them as well. Sure, people like citing themselves. Editors of double-blind journals already are supposed to desk reject work that says something like "in our prior study (Smith 2020)..." They should also desk reject obnoxiously self-referential citations --- like "Great advances have been made in method X (Smith 2016, 2017a, 2017b, 2018, 2019a, 2019b, 2020, 2021)"

micheles 4y ago

Having been on both sides of the peer review process in my field (Theoretical Physics), I say we should completely abolish it. Just publish the papers on the preprint archives. Some will be bad, some will be good, and you will know the impact of a paper after 10 years, nothing different from how it is now. Currently the very big majority of the effort in the reviews (on both sides of the process) is wasted time.