4 ms·

> Double-blind reviews are just peer-review theater. It is quite simple to guess which group the paper is from. It is difficult to guess the exact set of author

> Double-blind reviews are just peer-review theater. It is quite simple to guess which group the paper is from. It is difficult to guess the exact set of authors, but reviewers who are out to settle a score or to discard dismissively just need to know a ballpark of where the paper is from in order to stonewall with an irascible review. In some fields like AI knowing the exact authors is also quite common. Because the vast majority of researchers are employed by just a handful of big labs, the reviewers (and sometimes organizers) with experience in the subfield are employed by exactly the same lab(s). So they already know each other and each other's research anyway. Add to that that citations and prior work usually give away the set of authors too.

bo1024 4y ago
The opposite issue also - conflicts of interest can automatically disqualify reviewers, and if you are conflicted with all of microsoft research, or all of google research, etc., then the number of qualified reviewers shrinks dramatically.
- bumby 4y ago
  Who generally identifies these conflicts of interest? Is it reliant on self-reporting?
  - bo1024 4y ago
    In my experience, yes, essentially all self-reporting. They might do something like automatically conflict your email address domain (*.mit.edu), which is fun when random unrelated people sign up with gmail addresses.
godelski 4y ago
Not only this, but people publish their papers months ahead on arxiv. Big labs have big publicity and thousands will read their papers. They can often do more impactful work too because the massive amount of compute that they have. There's a good chance that if the reviewers are qualified to review the material they have already read the paper. (or a public version can be found) One thing I really wish is that we ensured that code works properly. I don't expect a full training but at least verify checkpoints.
- arinlen 4y ago
  > Not only this, but people publish their papers months ahead on arxiv. What's the difference between posting stuff on arxiv or uploading a pdf to your blog? > Big labs have big publicity and thousands will read their papers. Peer review is supposed to circumvent the problem of a marketing budget being a factor in whether your paper is deemed relevant or not.
  - godelski 4y ago
    > What's the difference between posting stuff on arxiv or uploading a pdf to your blog? There isn't. My point is that the papers aren't blind because they've been read before. Publishing onto your blog would have the same result. I'll note that this isn't common practice in other research communities. > Peer review is supposed to circumvent the problem of a marketing budget being a factor in whether your paper is deemed relevant or not. I don't see how this could even happen. Let's even say you are a no name lab and get accepted, you're in a list of hundreds or even a thousand other papers. There's publicity, yes, but it isn't massive. Let me put it this way. If you are a grad student at MIT or Stanford you will get hundreds of twitter followers relatively quickly. These followers are going to see you posting links to your papers. You have a larger network effect just by being at a larger and more prestigious school. That is the main advantage of prestigious schools and I don't think there's anything necessarily wrong with that. But we do have to admit that there are factors that are highly important here beyond merit based. Being at MIT or Stanford does not mean your paper has more merit than a no name school or lab. That no name lab will not have the same reach, which results in fewer citations (our main metric of relevancy), and fewer researchers will be able to expand upon the work. The major problem here is that we're pretending that this doesn't happen. I don't know how to solve this issue or even know if it can be solved. But it is really dumb to go around saying everything is merit based and not acknowledge the other factors at play here. It is pretending that there's a stronger signal in the noise than there is.

bo1024 4y ago

The opposite issue also - conflicts of interest can automatically disqualify reviewers, and if you are conflicted with all of microsoft research, or all of google research, etc., then the number of qualified reviewers shrinks dramatically.

bumby 4y ago
Who generally identifies these conflicts of interest? Is it reliant on self-reporting?
- bo1024 4y ago
  In my experience, yes, essentially all self-reporting. They might do something like automatically conflict your email address domain (*.mit.edu), which is fun when random unrelated people sign up with gmail addresses.

godelski 4y ago

Not only this, but people publish their papers months ahead on arxiv. Big labs have big publicity and thousands will read their papers. They can often do more impactful work too because the massive amount of compute that they have. There's a good chance that if the reviewers are qualified to review the material they have already read the paper. (or a public version can be found) One thing I really wish is that we ensured that code works properly. I don't expect a full training but at least verify checkpoints.

arinlen 4y ago
> Not only this, but people publish their papers months ahead on arxiv. What's the difference between posting stuff on arxiv or uploading a pdf to your blog? > Big labs have big publicity and thousands will read their papers. Peer review is supposed to circumvent the problem of a marketing budget being a factor in whether your paper is deemed relevant or not.
- godelski 4y ago
  > What's the difference between posting stuff on arxiv or uploading a pdf to your blog? There isn't. My point is that the papers aren't blind because they've been read before. Publishing onto your blog would have the same result. I'll note that this isn't common practice in other research communities. > Peer review is supposed to circumvent the problem of a marketing budget being a factor in whether your paper is deemed relevant or not. I don't see how this could even happen. Let's even say you are a no name lab and get accepted, you're in a list of hundreds or even a thousand other papers. There's publicity, yes, but it isn't massive. Let me put it this way. If you are a grad student at MIT or Stanford you will get hundreds of twitter followers relatively quickly. These followers are going to see you posting links to your papers. You have a larger network effect just by being at a larger and more prestigious school. That is the main advantage of prestigious schools and I don't think there's anything necessarily wrong with that. But we do have to admit that there are factors that are highly important here beyond merit based. Being at MIT or Stanford does not mean your paper has more merit than a no name school or lab. That no name lab will not have the same reach, which results in fewer citations (our main metric of relevancy), and fewer researchers will be able to expand upon the work. The major problem here is that we're pretending that this doesn't happen. I don't know how to solve this issue or even know if it can be solved. But it is really dumb to go around saying everything is merit based and not acknowledge the other factors at play here. It is pretending that there's a stronger signal in the noise than there is.