4 ms·

It would be a much better idea if double-blind peer review were a default. See "Nobel and Novice: Author Prominence Affects Peer Review": https://papers.ssrn.co

by Qem 4y ago

It would be a much better idea if double-blind peer review were a default. See "Nobel and Novice: Author Prominence Affects Peer Review": https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4190976 https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4190976

RcouF1uZ4gsC 4y ago
Double-blind peer review sounds nice in theory, but I don't think it would hold up in practice, especially in specialized domains. In those domains, everyone knows everyone else. Even if you hid the name, looking at the writing style, the subject, the choice of analysis software, the type of study. etc would largely give away who the authors are. For example I bet you could figure out that Richard Stallman wrote a particular paper, even if I removed his name from it. Or on HN, how many times have you read a comment and gone, "that sounds like..." even without fully noticing who posted it.
- gernb 4y ago
  ...
  - mturmon 4y ago
    You seem to be talking about a different kind of peer review.
- jiggawatts 4y ago
  Worse still, many papers are basically unintelligible to anyone outside of an "inner circle". Every time this is raised, inevitably people will pipe up and say "Oh, you're just not a <specialist in field>, that's why you can't read the paper!" For example, a new particle discovery paper was being debated in a forum and several people raised the issue that the paper was impenetrable, and was followed with the usual retorts of "you're not a particle physicist!". There was also a response from a particle physicist who works at CERN. He basically said that he himself could not understand the paper, despite working on a similar experiment on a different instrument. His office was down the hall from the office where the paper was written, but each team has their own terminology, jargon, and instrument-specific calibrations. Essentially, they can review their own papers, but nobody else can meaningfully validate it, or even understand it in a useful way! The problem with how modern science is performed is that it rewards the act of publishing a paper, not the utility, validity, or comprehensibility of the paper. In fact, writing papers to superficially look more 'serious' is rewarded, so papers that start off quite readable are often edited to be more obtuse and full of unnecessary symbols, terminology, or equations. I noticed this in Computer Science, where earlier papers in the field from the 60s and 70s were very readable, but then slowly started getting filled with Greek symbols and obscure terminology for no reason. There is no long history of Ancient Greek Computer Science that needs to be preserved for continuity with papers published hundreds of years ago! This is pure intellectual wankery, and has resulted in people like me unable to follow some modern CS papers on topics that I'm already familiar with. PS: Imagine the quality of "open source libraries" if everyone got accolades for publishing their code to GitHub, NuGet, or NPM, but no reward for anything else. No in-place updates. No direct usage. No pull requests. No feedback other than a one-time rubber stamp. No issue tracker. Nothing. Just... publish your code once and walk away, you get a cookie. Doesn't even matter if it compiles. Doesn't matter if it's pseudo-code you made up yourself. Just get on there, and you get something. But you get nothing for anything else. Picture that. That's modern science.
  - autoexec 4y ago
    > Worse still, many papers are basically unintelligible to anyone outside of an "inner circle". Right now we have papers simply filled with nonsense (https://link.springer.com/article/10.1007/s12517-021-08021-2 https://link.springer.com/article/10.1007/s12517-021-08021-2) and papers on midi-chlorians (https://www.livescience.com/59927-midi-chlorians-paper-accepted-by-journals.html https://www.livescience.com/59927-midi-chlorians-paper-accep...). Even without the needlessly over-complicated writing, some papers may really only be able to be meaningfully reviewed by a small number of experts, but that won't be a huge number. Really I'd be happy if we managed to catch the papers any child in grade school could spot as phony. Maybe we shouldn't be asking if peer-review is a good idea based on it's performance so far until we give it a serious effort.
  - hoosieree 4y ago
    These perverse incentives (publish or perish; using big words to appear more serious) are far bigger problems than peer review being single-blind or not. Yesterday I started looking for a token embedding method, and my search turned up word2vec and GLoVe. After skimming both papers I am leaning toward word2vec solely because it was easier for me to understand the big idea, even though GLoVe may perform better. Personally I would rather see code than a LaTeX-typeset equation. With typeset equations, the author has plausible deniability that you didn't interpret their notation just right, or oops that was supposed to be a boldface capital X, which implies a different data type. But if you provide code, anyone can run it and independently verify the results. Maybe the fact that no one can independently verify your equations/pseudocode is a feature, not a bug.
- ErikVandeWater 4y ago
  It might not be perfectly effective, but I don't see any substantive drawbacks vs the current system.
- pessimizer 4y ago
  It seems like in small fields, it would just be adding implausible deniability. Neutrality theater?
- mturmon 4y ago
  So, I could be charitably described as mid-career, and I had this discussion a few times a couple decades ago -- in which I took the position you have. It's not an unreasonable position. I was a PC member for what is now called NeurIPS, and I remember one particular disagreement with a conference co-chair who advocated blinded reviews. I have since come to believe that the above objections to blinding are mistaken, and that things should generally be moved to double-blind where at all possible. What happened is, I participated in blinded review processes (for proposals and for conferences) and they turned out to be different. Just that level of de-coupling -- where you aren't 100% sure that the author/proposer is who you suspect -- turns out to matter a great deal psychologically. It changes the framing. And more important, the results seem better.
LetThereBeLight 4y ago
Double-blind is not all that feasible unless preprints, conference posters, and talks were also all done anonymously.
- rovr138 4y ago
  Know of a few people that have gotten dismissive comments to review better the work from X paper of Y author... when the person submitting is that same one. You can't do much.
kentlyons 4y ago
The field(s) I'm in use double blind and it's broken for a host of other reasons. It's really hard to build off a string of work for example. Students (eg the ones writing papers) are often bad at the anonymization requirements. On the other side, the reviewers can often form an opinion on likely authors (whether correct or not). And then up one level (at the editorial / program committee level) that anonymization goes away since they need to check for reviewer conflicts, etc.
- culopatin 4y ago
  Also in fields small enough that everyone knows each other, you know exactly what the rest is working on because they collaborate often and there aren’t that many reviewers
jseliger 4y ago
That would be an improvement, but the formal peer-review process itself is a problem that can and should be solved: https://jakeseliger.com/2020/05/24/a-simple-solution-to-peer-review-problems/ https://jakeseliger.com/2020/05/24/a-simple-solution-to-peer...
Vinnl 4y ago
I don't think there's much research that double blind significantly improves the process. Been a while since I read it, but IIRC this article goes into that: https://absolutelymaybe.plos.org/2015/05/13/weighing-up-anonymity-and-openness-in-publication-peer-review/ https://absolutelymaybe.plos.org/2015/05/13/weighing-up-anon...
spoonjim 4y ago
Modern academics are so niche that people reviewing a paper can pretty much guess who wrote it if they are a known quantity.
- SassyThrowaway 4y ago
  Perhaps, surprisingly, it might go both ways: when I know who is on the PC, I think I can tell which person wrote which review, because I both know their work and met them personally. In fact, those people on the PC is why I am submitting to a venue in the first place, because I want reviewers who will easily understand my work.

by Qem 4y ago

RcouF1uZ4gsC 4y ago

Double-blind peer review sounds nice in theory, but I don't think it would hold up in practice, especially in specialized domains. In those domains, everyone knows everyone else. Even if you hid the name, looking at the writing style, the subject, the choice of analysis software, the type of study. etc would largely give away who the authors are. For example I bet you could figure out that Richard Stallman wrote a particular paper, even if I removed his name from it. Or on HN, how many times have you read a comment and gone, "that sounds like..." even without fully noticing who posted it.

gernb 4y ago
...
- mturmon 4y ago
  You seem to be talking about a different kind of peer review.
jiggawatts 4y ago
Worse still, many papers are basically unintelligible to anyone outside of an "inner circle". Every time this is raised, inevitably people will pipe up and say "Oh, you're just not a <specialist in field>, that's why you can't read the paper!" For example, a new particle discovery paper was being debated in a forum and several people raised the issue that the paper was impenetrable, and was followed with the usual retorts of "you're not a particle physicist!". There was also a response from a particle physicist who works at CERN. He basically said that he himself could not understand the paper, despite working on a similar experiment on a different instrument. His office was down the hall from the office where the paper was written, but each team has their own terminology, jargon, and instrument-specific calibrations. Essentially, they can review their own papers, but nobody else can meaningfully validate it, or even understand it in a useful way! The problem with how modern science is performed is that it rewards the act of publishing a paper, not the utility, validity, or comprehensibility of the paper. In fact, writing papers to superficially look more 'serious' is rewarded, so papers that start off quite readable are often edited to be more obtuse and full of unnecessary symbols, terminology, or equations. I noticed this in Computer Science, where earlier papers in the field from the 60s and 70s were very readable, but then slowly started getting filled with Greek symbols and obscure terminology for no reason. There is no long history of Ancient Greek Computer Science that needs to be preserved for continuity with papers published hundreds of years ago! This is pure intellectual wankery, and has resulted in people like me unable to follow some modern CS papers on topics that I'm already familiar with. PS: Imagine the quality of "open source libraries" if everyone got accolades for publishing their code to GitHub, NuGet, or NPM, but no reward for anything else. No in-place updates. No direct usage. No pull requests. No feedback other than a one-time rubber stamp. No issue tracker. Nothing. Just... publish your code once and walk away, you get a cookie. Doesn't even matter if it compiles. Doesn't matter if it's pseudo-code you made up yourself. Just get on there, and you get something. But you get nothing for anything else. Picture that. That's modern science.
- autoexec 4y ago
  > Worse still, many papers are basically unintelligible to anyone outside of an "inner circle". Right now we have papers simply filled with nonsense (https://link.springer.com/article/10.1007/s12517-021-08021-2 https://link.springer.com/article/10.1007/s12517-021-08021-2) and papers on midi-chlorians (https://www.livescience.com/59927-midi-chlorians-paper-accepted-by-journals.html https://www.livescience.com/59927-midi-chlorians-paper-accep...). Even without the needlessly over-complicated writing, some papers may really only be able to be meaningfully reviewed by a small number of experts, but that won't be a huge number. Really I'd be happy if we managed to catch the papers any child in grade school could spot as phony. Maybe we shouldn't be asking if peer-review is a good idea based on it's performance so far until we give it a serious effort.
- hoosieree 4y ago
  These perverse incentives (publish or perish; using big words to appear more serious) are far bigger problems than peer review being single-blind or not. Yesterday I started looking for a token embedding method, and my search turned up word2vec and GLoVe. After skimming both papers I am leaning toward word2vec solely because it was easier for me to understand the big idea, even though GLoVe may perform better. Personally I would rather see code than a LaTeX-typeset equation. With typeset equations, the author has plausible deniability that you didn't interpret their notation just right, or oops that was supposed to be a boldface capital X, which implies a different data type. But if you provide code, anyone can run it and independently verify the results. Maybe the fact that no one can independently verify your equations/pseudocode is a feature, not a bug.
ErikVandeWater 4y ago
It might not be perfectly effective, but I don't see any substantive drawbacks vs the current system.
pessimizer 4y ago
It seems like in small fields, it would just be adding implausible deniability. Neutrality theater?
mturmon 4y ago
So, I could be charitably described as mid-career, and I had this discussion a few times a couple decades ago -- in which I took the position you have. It's not an unreasonable position. I was a PC member for what is now called NeurIPS, and I remember one particular disagreement with a conference co-chair who advocated blinded reviews. I have since come to believe that the above objections to blinding are mistaken, and that things should generally be moved to double-blind where at all possible. What happened is, I participated in blinded review processes (for proposals and for conferences) and they turned out to be different. Just that level of de-coupling -- where you aren't 100% sure that the author/proposer is who you suspect -- turns out to matter a great deal psychologically. It changes the framing. And more important, the results seem better.

LetThereBeLight 4y ago

Double-blind is not all that feasible unless preprints, conference posters, and talks were also all done anonymously.

rovr138 4y ago
Know of a few people that have gotten dismissive comments to review better the work from X paper of Y author... when the person submitting is that same one. You can't do much.

kentlyons 4y ago

The field(s) I'm in use double blind and it's broken for a host of other reasons. It's really hard to build off a string of work for example. Students (eg the ones writing papers) are often bad at the anonymization requirements. On the other side, the reviewers can often form an opinion on likely authors (whether correct or not). And then up one level (at the editorial / program committee level) that anonymization goes away since they need to check for reviewer conflicts, etc.

culopatin 4y ago
Also in fields small enough that everyone knows each other, you know exactly what the rest is working on because they collaborate often and there aren’t that many reviewers

jseliger 4y ago

That would be an improvement, but the formal peer-review process itself is a problem that can and should be solved: https://jakeseliger.com/2020/05/24/a-simple-solution-to-peer-review-problems/ https://jakeseliger.com/2020/05/24/a-simple-solution-to-peer...

Vinnl 4y ago

I don't think there's much research that double blind significantly improves the process. Been a while since I read it, but IIRC this article goes into that: https://absolutelymaybe.plos.org/2015/05/13/weighing-up-anonymity-and-openness-in-publication-peer-review/ https://absolutelymaybe.plos.org/2015/05/13/weighing-up-anon...

spoonjim 4y ago

Modern academics are so niche that people reviewing a paper can pretty much guess who wrote it if they are a known quantity.

SassyThrowaway 4y ago
Perhaps, surprisingly, it might go both ways: when I know who is on the PC, I think I can tell which person wrote which review, because I both know their work and met them personally. In fact, those people on the PC is why I am submitting to a venue in the first place, because I want reviewers who will easily understand my work.