5 ms·

As someone who is inside the academic publication loop, I am in favor of double-blind, non-public peer reviews because I don't want to invite 4Chan into my dail

As someone who is inside the academic publication loop, I am in favor of double-blind, non-public peer reviews because I don't want to invite 4Chan into my daily work. I believe that double-blind, non-public reviews do a great job at protecting those in a more precarious situation: grad students are not judged by their lack of publication record, there is no permanent public record of all those times they got rejected, and reviewers can be both more honest and more certain that a rejection won't lead to a 4Chan/Twitter mob coming for them. According to the ACL's 2019 survey [1], "female respondents were less likely to support public review than male respondents". I'd be weary of implementing any changed that would make academia even more hostile to women and minorities. [1] http://acl2019pcblog.fileli.unipi.it/wp-content/uploads/2019/07/ReportACL2019ReviewingSurvey.pdf http://acl2019pcblog.fileli.unipi.it/wp-content/uploads/2019...

LudwigNagasena 4y ago
Interesting to see how the good ol’ “think of the children” defense has morphed into “think of the women and minorities”.
jhbadger 4y ago
I just don't see how double-blind peer review can work. To properly peer review a manuscript you often have to read the papers cited (because often details of the methods of the current manuscript are described in previously published work). These citations are almost always to the authors' own work. So it is trivial for any peer reviewer to figure out who wrote the manuscript they are reviewing, even if their names are blinded.
- humanistbot 4y ago
  I hear this a lot and I just don't buy it. The whole idea of science is that you can describe methods in text, send that text around the world, and have other people use those same methods to do their own independent studies. So even if you know that lab X really likes using a particular method, it could be someone from lab Y who read the papers from lab X and wanted to use them as well. Sure, people like citing themselves. Editors of double-blind journals already are supposed to desk reject work that says something like "in our prior study (Smith 2020)..." They should also desk reject obnoxiously self-referential citations --- like "Great advances have been made in method X (Smith 2016, 2017a, 2017b, 2018, 2019a, 2019b, 2020, 2021)"
  - maegul 4y ago
    I'm sure different fields have different dynamics, and in some what you describe here might be viable and accurate. But my own impression and experience is that your statement is naive. The key thing I think you miss is that researchers/labs/groups develop fairly specific interests, expertise and "research programs". So taking any paper and looking at the specific area or phenomenon being probed, the techniques/methods used, and the questions or interests driving the research and its analysis, and you'll get a pretty unique "fingerprint" of the researchers that anyone in the same field should be able to identify with pretty good accuracy. The naivety of your statement I think is this idea that experiments or acts of research are kind of plucked from the air by any who are interested and then the method or technique required is simply downloaded and executed like software. Researchers aren't generic like computing hardware tries to be. Far from it. A researcher and their research or ability to do research is much more like a tree ... grown organically through experience and pretty fixed and inflexible once it's mature. None of this is to be really critical of your statement. In fact, I think it's very useful (provided I'm onto something and not merely bitter) to state the ideal practice of science and contrast it with reality. That we can't do double-blind review is almost certainly a bad thing. And it also tells us something about what kind of "science" (here the global collective enterprise) we have. I think I'm correct about the "fixation" of researchers. But I also think that the culture of science we have encourages this process. I've seen it myself: researchers trying a minor change in field and technique (very minor) and hitting massive resistance from the funding agencies. The conclusion we all drew from seeing it was that the techniques you can get funding for are mostly fixed by your first 10 papers or so. This could be changed and research could get closer to what you describe and I think that would be a good thing. But, as I stated in another comment, to get there requires change beyond the peer review process. Something else it tells us is that our science operates in a relatively closed and inward looking cultural ecosystem, which indicates to me some degree of corruption is almost guaranteed. If we don't have good diversity and competitive and corrective interactions across space/culture, then, as the "coffins in the ground" adage tells us, we have to rely on diversity across time/generations. And indeed researchers know this just from experience. Someone winning a Nobel prize is like carving their thought in stone ... it takes time for us to move on. An interesting study in this regard might science during the USSR. I've read glancingly that mathematics research progressed differently between the West and USSR and that, for this interested, cross-pollination was fruitful. An analysis of this could prove insightful. I wonder whether something similar is going on in China at the moment.
q-big 4y ago
> I am in favor of double-blind, non-public peer reviews because I don't want to invite 4Chan into my daily work. At least for mathematics papers, the community who works in a particular area, and is thus actually able to do a review the paper, is pretty small. Since, on the other hand, the style that you use for mathematical proofs is like a fingerprint, double-blind peer reviews are next to impossible.

LudwigNagasena 4y ago

Interesting to see how the good ol’ “think of the children” defense has morphed into “think of the women and minorities”.

jhbadger 4y ago

I just don't see how double-blind peer review can work. To properly peer review a manuscript you often have to read the papers cited (because often details of the methods of the current manuscript are described in previously published work). These citations are almost always to the authors' own work. So it is trivial for any peer reviewer to figure out who wrote the manuscript they are reviewing, even if their names are blinded.

humanistbot 4y ago
I hear this a lot and I just don't buy it. The whole idea of science is that you can describe methods in text, send that text around the world, and have other people use those same methods to do their own independent studies. So even if you know that lab X really likes using a particular method, it could be someone from lab Y who read the papers from lab X and wanted to use them as well. Sure, people like citing themselves. Editors of double-blind journals already are supposed to desk reject work that says something like "in our prior study (Smith 2020)..." They should also desk reject obnoxiously self-referential citations --- like "Great advances have been made in method X (Smith 2016, 2017a, 2017b, 2018, 2019a, 2019b, 2020, 2021)"
- maegul 4y ago
  I'm sure different fields have different dynamics, and in some what you describe here might be viable and accurate. But my own impression and experience is that your statement is naive. The key thing I think you miss is that researchers/labs/groups develop fairly specific interests, expertise and "research programs". So taking any paper and looking at the specific area or phenomenon being probed, the techniques/methods used, and the questions or interests driving the research and its analysis, and you'll get a pretty unique "fingerprint" of the researchers that anyone in the same field should be able to identify with pretty good accuracy. The naivety of your statement I think is this idea that experiments or acts of research are kind of plucked from the air by any who are interested and then the method or technique required is simply downloaded and executed like software. Researchers aren't generic like computing hardware tries to be. Far from it. A researcher and their research or ability to do research is much more like a tree ... grown organically through experience and pretty fixed and inflexible once it's mature. None of this is to be really critical of your statement. In fact, I think it's very useful (provided I'm onto something and not merely bitter) to state the ideal practice of science and contrast it with reality. That we can't do double-blind review is almost certainly a bad thing. And it also tells us something about what kind of "science" (here the global collective enterprise) we have. I think I'm correct about the "fixation" of researchers. But I also think that the culture of science we have encourages this process. I've seen it myself: researchers trying a minor change in field and technique (very minor) and hitting massive resistance from the funding agencies. The conclusion we all drew from seeing it was that the techniques you can get funding for are mostly fixed by your first 10 papers or so. This could be changed and research could get closer to what you describe and I think that would be a good thing. But, as I stated in another comment, to get there requires change beyond the peer review process. Something else it tells us is that our science operates in a relatively closed and inward looking cultural ecosystem, which indicates to me some degree of corruption is almost guaranteed. If we don't have good diversity and competitive and corrective interactions across space/culture, then, as the "coffins in the ground" adage tells us, we have to rely on diversity across time/generations. And indeed researchers know this just from experience. Someone winning a Nobel prize is like carving their thought in stone ... it takes time for us to move on. An interesting study in this regard might science during the USSR. I've read glancingly that mathematics research progressed differently between the West and USSR and that, for this interested, cross-pollination was fruitful. An analysis of this could prove insightful. I wonder whether something similar is going on in China at the moment.

q-big 4y ago

> I am in favor of double-blind, non-public peer reviews because I don't want to invite 4Chan into my daily work. At least for mathematics papers, the community who works in a particular area, and is thus actually able to do a review the paper, is pretty small. Since, on the other hand, the style that you use for mathematical proofs is like a fingerprint, double-blind peer reviews are next to impossible.