5 ms·

I've been in academic research for 22 years. Based on my experience, I'll go out on a limb and say that even single-blind reviews are to be discarded. This is

by sn41 4y ago

I've been in academic research for 22 years. Based on my experience, I'll go out on a limb and say that even single-blind reviews are to be discarded. This is leading to quite irresponsible reviews. Instead, authored and credited reviews might lead to more responsible reviews, or reviewers respectfully declining when they might not know the topic. Instead, in CS, there is a tendency to hide behind an abrasive negative review when the reality is that the reviewer does not understand the paper. Programme Committees are relieved to find a negative review, however unfair or off-kilter it is, because more rejected papers will decrease the acceptance ratio of the conference, hence make it appear more competitive. Double-blind reviews are just peer-review theater. It is quite simple to guess which group the paper is from. It is difficult to guess the exact set of authors, but reviewers who are out to settle a score or to discard dismissively just need to know a ballpark of where the paper is from in order to stonewall with an irascible review.

Gimpei 4y ago
The one experience I had with peer review before leaving academia was actually pretty nice. One of the reviewers was a bit of a hardass, but they made good points and I think made the paper much better. As a consumer of papers now, I find peer review pretty useful. I can’t read everything that comes out. Even if I just read abstracts, I need some sort of filter. Publication in a top tier journal raises the probability that egregious errors haven’t been made and that the paper is worthy of my time.
- sn41 4y ago
  Peer review is useful, no doubt. It is only reviewer anonymity that probably should be re-examined.
  - bumby 4y ago
    There would probably be a need for a mechanism to ensure objectivity. E.g., if Reviewer A rejects Reviewer B, there may be a higher chance Reviewer B will reject Review A's future work, just out of spite.
cirgue 4y ago
I have no serious involvement in academia. What is the proportion of work that gets reviewed by someone with an axe to grind? I don’t doubt that it happens regularly but I have no conception of the magnitude of the problem.
- nextos 4y ago
  In experimental fields, when submitting to Nature / Science / Cell, it's likely that your work ends up getting reviewed by someone who sees you as a competitor (this mentality drives me mad). Hence, they will sometimes try to make sure you don't publish or at least get downgraded to sister journals, such as Nature Communications. Another problem are editorial decisions. Great work is often desk rejected because it's judged not to be significant or interersting enough (usually the author comes from a less prominent university). For example, the original CRISPR article was desk rejected by most journals and the fact that the author's affiliation was a no-name research centre in Alicante (Spain) surely played a role. This delayed CRISPR getting widely known by a few years...
  - SlickNixon 4y ago
    >For example, the original CRISPR article was desk rejected by most journals and the fact that the author's affiliation was a no-name research centre in Alicante (Spain) surely played a role. This delayed CRISPR getting widely known by a few years... This is interesting. Has there been a write up on this saga anywhere?
    - nextos 4y ago
      I can't find better writeups at the minute, but this is a good starting point: https://en.wikipedia.org/wiki/Francisco_Mojica#CRISPR_as_a_microbial_immune_system https://en.wikipedia.org/wiki/Francisco_Mojica#CRISPR_as_a_m...
- gjulianm 4y ago
  In my experience, about 30-40% of the reviews I've seen are a mix between "axe to grind" and "I have no idea what I'm talking about". Some are extremely blatant, such as reviewers "suggesting" you should cite a certain paper that isn't really related to your work. Others are very strict with the "this is just practical knowledge so it doesn't advance the science", and then there's the set of "I didn't understand the paper so I'm going to criticize it based on my misconceptions of it and reject it". I'm so glad to be out of that world, to be honest.
mudrockbestgirl 4y ago
> Double-blind reviews are just peer-review theater. It is quite simple to guess which group the paper is from. It is difficult to guess the exact set of authors, but reviewers who are out to settle a score or to discard dismissively just need to know a ballpark of where the paper is from in order to stonewall with an irascible review. In some fields like AI knowing the exact authors is also quite common. Because the vast majority of researchers are employed by just a handful of big labs, the reviewers (and sometimes organizers) with experience in the subfield are employed by exactly the same lab(s). So they already know each other and each other's research anyway. Add to that that citations and prior work usually give away the set of authors too.
- bo1024 4y ago
  The opposite issue also - conflicts of interest can automatically disqualify reviewers, and if you are conflicted with all of microsoft research, or all of google research, etc., then the number of qualified reviewers shrinks dramatically.
  - bumby 4y ago
    Who generally identifies these conflicts of interest? Is it reliant on self-reporting?
    - bo1024 4y ago
      In my experience, yes, essentially all self-reporting. They might do something like automatically conflict your email address domain (*.mit.edu), which is fun when random unrelated people sign up with gmail addresses.
- godelski 4y ago
  Not only this, but people publish their papers months ahead on arxiv. Big labs have big publicity and thousands will read their papers. They can often do more impactful work too because the massive amount of compute that they have. There's a good chance that if the reviewers are qualified to review the material they have already read the paper. (or a public version can be found) One thing I really wish is that we ensured that code works properly. I don't expect a full training but at least verify checkpoints.
maegul 4y ago
Agree. I used to think double blind was an obvious requirement for a good peer review system. Your “theatre” take is aptly harsh I’d say. Also hard agree on publishing reviews. It seems to be the only viable modification of peer review that we have. My latest experience with peer review was just bad. No matter the conclusion they drew, there was just a clear lack of quality and understanding in their thoughts and effort. And this is not infrequent. More generally, such a modification has knock on effects that probably warrant some thought on what we do with our publication practices. Published reviews will probably require more work from reviewers who are already time poor. Better reviews, hopefully, but fewer of them and fewer papers? Counting a good review as a citable publication might be worth considering, as essentially mini idiosyncratic literature reviews. Perhaps in combination with Registered Reports[1], where the proposal is published and reviewed before the study/work is done, in which case the reviewer is closer to symbiotic co-publisher. Either way, it seems that for both authors and reviewers, who are effectively the same people and both rely on peer-review as a guarantee of the value of their work, that the actual work of peer review needs to be taken more seriously and not conceived of as some sort of feudal aristocratic gentlemen’s duty. ~~~~~~~~ [1] https://www.cos.io/initiatives/registered-reports https://www.cos.io/initiatives/registered-reports
Helmut10001 4y ago
> Instead, in CS, there is a tendency to hide behind an abrasive negative review when the reality is that the reviewer does not understand the paper. Never heard this argument, but having been under many double blind peer reviews in the last 10 years, I can sign this. Interestingly, in my last paper for PLOS One, one of two reviewers chose to lift anonymity and specifically his comments were on-focus, supportive and substantial.
LudwigNagasena 4y ago
I really hope that something like https://openreview.net/ https://openreview.net/ becomes the norm for all scientific journals in near future. The open discussions not only benefit the peer review process but also act as context and learning material for others.
auggierose 4y ago
Yes, remove all the blinders. Of course, in practice, this has the problem that it is already hard to get reviewers, because nobody has time anymore. Removing the anonymity of reviewers leads to the judge becoming open to judgement themselves, and many would not like that. Furthermore, there might be severe repercussions if the reviewed is more powerful than the reviewer, especially in authoritarian societies. But maybe one should just exclude these societies from peer review, and let them do their own thing. I think peer review has to become a market place. Let everyone choose for themselves which paper they would like to review, and there is both positive and negative credit for both reviewers and authors. The pool of reviewers of a paper shares 25% of the credit that the authors of the paper get. This way you are incentivised to review even outlier papers, because if they are successful you get a substantial amount of credit for them. How the credit is distributed among the reviewers should also incorporate a time factor, so that reviewers rushing to a paper that is already successful don't get nearly as much credit as a lone early reviewer.
- collyw 4y ago
  Peer review is done for free, maybe it needs to be compensated.
  - auggierose 4y ago
    That's why I use the term "credit". It needs to have real-world value of some sort. I don't think just paying people is the solution, that would just mean the richer you are, the more reviews you get, and the better your reviews are. You can argue they should be paid via a neutral source of money, but there is no such thing.
    - suoduandao2 4y ago
      If the problem of gaming the system could be gotten around, something very similar to internet comments could work - anyone can post a public review of anyone's work, in which case reputation would become the currency. Successfully criticizing someone with higher reputation than yourself would gain more status than praising them, but also come with more risk. It would also make it a lot less meaningful to get those 'not new ideas lol' comments that others on this thread have complained of. I quite like the fact that whatever someone says on the internet now, the first comment is usually the best-argued counterpoint. It seems like a dynamic that could be brought to science quite well. I may not have the time to parse an entire paper in a field unfamiliar to me, but if I come across it and the top comment is 'there's a math error in table 6 that throws paragraph 4 into doubt', I can certainly verify that for myself. That sort of 'peer' review would make science much more accessible to the interested layperson.
spoonjim 4y ago
All of the process creates incredible gatekeeping effects. Science would be much more robust if published github-style: post your results, engage with the community, fix issues, respond to critics, and eventually wrap it up, but always have the priority date of your original work recorded so that there's no dispute as to who invented it first.
arinlen 4y ago
> This is leading to quite irresponsible reviews. Instead, authored and credited reviews might lead to more responsible reviews, or reviewers respectfully declining when they might not know the topic. That misses the whole point of anonymous reviews, or even anonymous voting. You want your work to stand by itself. Otherwise you're blasting open a very corrupt door where appeals to authority, petty politics, careerism, and funding play a role in how a paper is approved. > Instead, in CS, there is a tendency to hide behind an abrasive negative review when the reality is that the reviewer does not understand the paper. Isn't the whole point of a paper to present a topic in a clear and understandable way, so that your peers are able to cut through the bullshit? If you pick a journal to publish your paper, which means you explicitly want the paper's editors to go through each and every single line of text you wrote to poke holes, but in the end once those holes are picked you complain that the journal you picked is not the right one for your flawless paper and that they all suck and their problem is that they don't understand your genius, what does this say about you and your work? When you pick a journal you pick the subset of your peers to review your work. If your peers point out problems then why not listen to them? Your comment reads a whole lot like "the fox and the grapes". https://en.wikipedia.org/wiki/The_Fox_and_the_Grapes https://en.wikipedia.org/wiki/The_Fox_and_the_Grapes
- auggierose 4y ago
  First, journals and conferences are different. I would expect more valuable feedback from journals than from conference reviews. Second, journals are overloaded, the journal reviewers unpaid, and frankly, they are not doing a good job anymore in reviewing work. Feedback that amounts to "maybe submit first to a conference" is not that valuable. Effectively, you have to have the idea, then execute it perfectly, then do a major marketing effort to popularise your work, and why exactly would you at the end of this still submit to a journal? Given that the journal either wants the rights to your work, or charges you for the pleasure of publishing with them. At this point, a journal is just another marketing step, and for marketing you pay, I guess.
- CrazyStat 4y ago
  > You want your work to stand by itself. Otherwise you're blasting open a very corrupt door where appeals to authority, petty politics, careerism, and funding play a role in how a paper is approved. This door is already open, and impossible to close [1]. For any reviewer closely familiar with the area the paper is in it is trivial to guess which group the paper came out of 9 times out of 10. Style, approach, citations, and just knowing what the other people in your area are working on all contribute. To achieve any sort of blindness you have to go to reviewers who don't know that area as well... which, I suppose, does decrease the potential for corruption--at the cost of the reviewers not understanding the paper as well. > Isn't the whole point of a paper to present a topic in a clear and understandable way, so that your peers are able to cut through the bullshit? Even a well written paper often requires some effort to understand, due to conceptual novelties, mathematical complexities, etc. Unfortunately there's little incentive for reviewers to put in much mental effort. Some of them will. Many won't. > When you pick a journal you pick the subset of your peers to review your work. If your peers point out problems then why not listen to them? Because the "problems" are often bullshit. You're assuming the reviewers are acting in good faith and finding legitimate problems with the paper, but they're often not--usually not out of malice, just laziness. There's no incentive to put any effort into a review above the minimum to convince the editor that your review is plausible. This doesn't take much since the editor is probably only going to glance briefly at the reviews. If you can finish your review in an hour instead of a day you can spend the rest of the day writing your own papers or doing other work that actually furthers your career. [1] In all but the largest, most active areas, where there are so many people working that nobody can keep up.
jltsiren 4y ago
CS publication culture is a large part of the problem. The average CS paper is small (in terms of person-years put into the project), so people submit many papers. Most papers are submitted to conferences first, which usually accept or reject after a single round of reviews. When the typical outcome is rejection and resubmission to another conference, the negative aspects of peer review become prominent. Over the years, I've drifted to the biological side of bioinformatics methods. The papers I write are similar to CS papers, but the projects are bigger, and I no longer submit papers to CS conferences. My typical experience with peer review is accept after 1-3 rounds of revisions. Hence I get to see more of the positive side, where the reviewers act as editors and their suggestions improve the paper. Of course, it's possible to get a CS-like experience with journals by being ambitious. If you try your luck by submitting the paper to prestigious journals it's unlikely to get in, your typical outcome is a desk rejection or rejection after a single round of reviews. Then you get to see more of the negative side of peer review. But I'm just not interested in playing that game.
- sn41 4y ago
  This is a very thoughtful take on the issue. My personal experience corroborates with this - CS conferences have more of these culture issues than mathematics journals, where the reviews tend to more thoughtful, and at the same time, considerate.
Archelaos 4y ago
> Instead, in CS, there is a tendency to hide behind an abrasive negative review when the reality is that the reviewer does not understand the paper. This problem goes even deeper, because it not only applies to papers but also to research grant applications. I even had once got a (positive) recommendation for one of my applications, but I could see from the statement that the reviewer did not really understand what was important.
- bumby 4y ago
  Not meant in a snarky way, but that almost feels to me as if the process worked. If they application committee did not see the value in the proposed work, it indicates they probably don't have much need for it. If they don't have much need, they shouldn't fund it. That's not to say the proposal isn't worthwhile, just that it didn't find the right audience. A contractor may have a whiz-bang new method of snow removal, but it doesn't mean people in Phoenix will want it (or even understand it).
hooby 4y ago
Just one question about that - what you are describing here - is that: .) an unsolvable problem inherent to all peer reviews? .) a problem that comes from specific practices, and could be solved by doing peer reviews differently? I'm just a layman - but if reviews are not done by peers (= experts working in the same field)... who else could properly review a paper?
- jraph 4y ago
  Your parent is not arguing against peer reviews, but against blind and double blind reviews. They are actually proposing that peer reviews are credited: > Instead, authored and credited reviews might lead to more responsible reviews, or reviewers respectfully declining when they might not know the topic. And yes, I do think reviewers should be credited. Because it's work, for one, that needed to be done so the paper could be published, and because it would probably indeed incentivize reviewers to do the work correctly. However, it could incentivize them to reject more papers, in fear of accepting a bad paper and be publicly advertised as a reviewer who reviewed and accepted it.

by sn41 4y ago

Gimpei 4y ago

The one experience I had with peer review before leaving academia was actually pretty nice. One of the reviewers was a bit of a hardass, but they made good points and I think made the paper much better. As a consumer of papers now, I find peer review pretty useful. I can’t read everything that comes out. Even if I just read abstracts, I need some sort of filter. Publication in a top tier journal raises the probability that egregious errors haven’t been made and that the paper is worthy of my time.

sn41 4y ago
Peer review is useful, no doubt. It is only reviewer anonymity that probably should be re-examined.
- bumby 4y ago
  There would probably be a need for a mechanism to ensure objectivity. E.g., if Reviewer A rejects Reviewer B, there may be a higher chance Reviewer B will reject Review A's future work, just out of spite.

cirgue 4y ago

I have no serious involvement in academia. What is the proportion of work that gets reviewed by someone with an axe to grind? I don’t doubt that it happens regularly but I have no conception of the magnitude of the problem.

nextos 4y ago
In experimental fields, when submitting to Nature / Science / Cell, it's likely that your work ends up getting reviewed by someone who sees you as a competitor (this mentality drives me mad). Hence, they will sometimes try to make sure you don't publish or at least get downgraded to sister journals, such as Nature Communications. Another problem are editorial decisions. Great work is often desk rejected because it's judged not to be significant or interersting enough (usually the author comes from a less prominent university). For example, the original CRISPR article was desk rejected by most journals and the fact that the author's affiliation was a no-name research centre in Alicante (Spain) surely played a role. This delayed CRISPR getting widely known by a few years...
- SlickNixon 4y ago
  >For example, the original CRISPR article was desk rejected by most journals and the fact that the author's affiliation was a no-name research centre in Alicante (Spain) surely played a role. This delayed CRISPR getting widely known by a few years... This is interesting. Has there been a write up on this saga anywhere?
  - nextos 4y ago
    I can't find better writeups at the minute, but this is a good starting point: https://en.wikipedia.org/wiki/Francisco_Mojica#CRISPR_as_a_microbial_immune_system https://en.wikipedia.org/wiki/Francisco_Mojica#CRISPR_as_a_m...
gjulianm 4y ago
In my experience, about 30-40% of the reviews I've seen are a mix between "axe to grind" and "I have no idea what I'm talking about". Some are extremely blatant, such as reviewers "suggesting" you should cite a certain paper that isn't really related to your work. Others are very strict with the "this is just practical knowledge so it doesn't advance the science", and then there's the set of "I didn't understand the paper so I'm going to criticize it based on my misconceptions of it and reject it". I'm so glad to be out of that world, to be honest.

mudrockbestgirl 4y ago

> Double-blind reviews are just peer-review theater. It is quite simple to guess which group the paper is from. It is difficult to guess the exact set of authors, but reviewers who are out to settle a score or to discard dismissively just need to know a ballpark of where the paper is from in order to stonewall with an irascible review. In some fields like AI knowing the exact authors is also quite common. Because the vast majority of researchers are employed by just a handful of big labs, the reviewers (and sometimes organizers) with experience in the subfield are employed by exactly the same lab(s). So they already know each other and each other's research anyway. Add to that that citations and prior work usually give away the set of authors too.

bo1024 4y ago
The opposite issue also - conflicts of interest can automatically disqualify reviewers, and if you are conflicted with all of microsoft research, or all of google research, etc., then the number of qualified reviewers shrinks dramatically.
- bumby 4y ago
  Who generally identifies these conflicts of interest? Is it reliant on self-reporting?
  - bo1024 4y ago
    In my experience, yes, essentially all self-reporting. They might do something like automatically conflict your email address domain (*.mit.edu), which is fun when random unrelated people sign up with gmail addresses.
godelski 4y ago
Not only this, but people publish their papers months ahead on arxiv. Big labs have big publicity and thousands will read their papers. They can often do more impactful work too because the massive amount of compute that they have. There's a good chance that if the reviewers are qualified to review the material they have already read the paper. (or a public version can be found) One thing I really wish is that we ensured that code works properly. I don't expect a full training but at least verify checkpoints.

maegul 4y ago

Agree. I used to think double blind was an obvious requirement for a good peer review system. Your “theatre” take is aptly harsh I’d say. Also hard agree on publishing reviews. It seems to be the only viable modification of peer review that we have. My latest experience with peer review was just bad. No matter the conclusion they drew, there was just a clear lack of quality and understanding in their thoughts and effort. And this is not infrequent. More generally, such a modification has knock on effects that probably warrant some thought on what we do with our publication practices. Published reviews will probably require more work from reviewers who are already time poor. Better reviews, hopefully, but fewer of them and fewer papers? Counting a good review as a citable publication might be worth considering, as essentially mini idiosyncratic literature reviews. Perhaps in combination with Registered Reports[1], where the proposal is published and reviewed before the study/work is done, in which case the reviewer is closer to symbiotic co-publisher. Either way, it seems that for both authors and reviewers, who are effectively the same people and both rely on peer-review as a guarantee of the value of their work, that the actual work of peer review needs to be taken more seriously and not conceived of as some sort of feudal aristocratic gentlemen’s duty. ~~~~~~~~ [1] https://www.cos.io/initiatives/registered-reports https://www.cos.io/initiatives/registered-reports

Helmut10001 4y ago

> Instead, in CS, there is a tendency to hide behind an abrasive negative review when the reality is that the reviewer does not understand the paper. Never heard this argument, but having been under many double blind peer reviews in the last 10 years, I can sign this. Interestingly, in my last paper for PLOS One, one of two reviewers chose to lift anonymity and specifically his comments were on-focus, supportive and substantial.

LudwigNagasena 4y ago

I really hope that something like https://openreview.net/ https://openreview.net/ becomes the norm for all scientific journals in near future. The open discussions not only benefit the peer review process but also act as context and learning material for others.

auggierose 4y ago

Yes, remove all the blinders. Of course, in practice, this has the problem that it is already hard to get reviewers, because nobody has time anymore. Removing the anonymity of reviewers leads to the judge becoming open to judgement themselves, and many would not like that. Furthermore, there might be severe repercussions if the reviewed is more powerful than the reviewer, especially in authoritarian societies. But maybe one should just exclude these societies from peer review, and let them do their own thing. I think peer review has to become a market place. Let everyone choose for themselves which paper they would like to review, and there is both positive and negative credit for both reviewers and authors. The pool of reviewers of a paper shares 25% of the credit that the authors of the paper get. This way you are incentivised to review even outlier papers, because if they are successful you get a substantial amount of credit for them. How the credit is distributed among the reviewers should also incorporate a time factor, so that reviewers rushing to a paper that is already successful don't get nearly as much credit as a lone early reviewer.

collyw 4y ago
Peer review is done for free, maybe it needs to be compensated.
- auggierose 4y ago
  That's why I use the term "credit". It needs to have real-world value of some sort. I don't think just paying people is the solution, that would just mean the richer you are, the more reviews you get, and the better your reviews are. You can argue they should be paid via a neutral source of money, but there is no such thing.
  - suoduandao2 4y ago
    If the problem of gaming the system could be gotten around, something very similar to internet comments could work - anyone can post a public review of anyone's work, in which case reputation would become the currency. Successfully criticizing someone with higher reputation than yourself would gain more status than praising them, but also come with more risk. It would also make it a lot less meaningful to get those 'not new ideas lol' comments that others on this thread have complained of. I quite like the fact that whatever someone says on the internet now, the first comment is usually the best-argued counterpoint. It seems like a dynamic that could be brought to science quite well. I may not have the time to parse an entire paper in a field unfamiliar to me, but if I come across it and the top comment is 'there's a math error in table 6 that throws paragraph 4 into doubt', I can certainly verify that for myself. That sort of 'peer' review would make science much more accessible to the interested layperson.

spoonjim 4y ago

All of the process creates incredible gatekeeping effects. Science would be much more robust if published github-style: post your results, engage with the community, fix issues, respond to critics, and eventually wrap it up, but always have the priority date of your original work recorded so that there's no dispute as to who invented it first.

arinlen 4y ago

> This is leading to quite irresponsible reviews. Instead, authored and credited reviews might lead to more responsible reviews, or reviewers respectfully declining when they might not know the topic. That misses the whole point of anonymous reviews, or even anonymous voting. You want your work to stand by itself. Otherwise you're blasting open a very corrupt door where appeals to authority, petty politics, careerism, and funding play a role in how a paper is approved. > Instead, in CS, there is a tendency to hide behind an abrasive negative review when the reality is that the reviewer does not understand the paper. Isn't the whole point of a paper to present a topic in a clear and understandable way, so that your peers are able to cut through the bullshit? If you pick a journal to publish your paper, which means you explicitly want the paper's editors to go through each and every single line of text you wrote to poke holes, but in the end once those holes are picked you complain that the journal you picked is not the right one for your flawless paper and that they all suck and their problem is that they don't understand your genius, what does this say about you and your work? When you pick a journal you pick the subset of your peers to review your work. If your peers point out problems then why not listen to them? Your comment reads a whole lot like "the fox and the grapes". https://en.wikipedia.org/wiki/The_Fox_and_the_Grapes https://en.wikipedia.org/wiki/The_Fox_and_the_Grapes

auggierose 4y ago
First, journals and conferences are different. I would expect more valuable feedback from journals than from conference reviews. Second, journals are overloaded, the journal reviewers unpaid, and frankly, they are not doing a good job anymore in reviewing work. Feedback that amounts to "maybe submit first to a conference" is not that valuable. Effectively, you have to have the idea, then execute it perfectly, then do a major marketing effort to popularise your work, and why exactly would you at the end of this still submit to a journal? Given that the journal either wants the rights to your work, or charges you for the pleasure of publishing with them. At this point, a journal is just another marketing step, and for marketing you pay, I guess.
CrazyStat 4y ago
> You want your work to stand by itself. Otherwise you're blasting open a very corrupt door where appeals to authority, petty politics, careerism, and funding play a role in how a paper is approved. This door is already open, and impossible to close [1]. For any reviewer closely familiar with the area the paper is in it is trivial to guess which group the paper came out of 9 times out of 10. Style, approach, citations, and just knowing what the other people in your area are working on all contribute. To achieve any sort of blindness you have to go to reviewers who don't know that area as well... which, I suppose, does decrease the potential for corruption--at the cost of the reviewers not understanding the paper as well. > Isn't the whole point of a paper to present a topic in a clear and understandable way, so that your peers are able to cut through the bullshit? Even a well written paper often requires some effort to understand, due to conceptual novelties, mathematical complexities, etc. Unfortunately there's little incentive for reviewers to put in much mental effort. Some of them will. Many won't. > When you pick a journal you pick the subset of your peers to review your work. If your peers point out problems then why not listen to them? Because the "problems" are often bullshit. You're assuming the reviewers are acting in good faith and finding legitimate problems with the paper, but they're often not--usually not out of malice, just laziness. There's no incentive to put any effort into a review above the minimum to convince the editor that your review is plausible. This doesn't take much since the editor is probably only going to glance briefly at the reviews. If you can finish your review in an hour instead of a day you can spend the rest of the day writing your own papers or doing other work that actually furthers your career. [1] In all but the largest, most active areas, where there are so many people working that nobody can keep up.

jltsiren 4y ago

CS publication culture is a large part of the problem. The average CS paper is small (in terms of person-years put into the project), so people submit many papers. Most papers are submitted to conferences first, which usually accept or reject after a single round of reviews. When the typical outcome is rejection and resubmission to another conference, the negative aspects of peer review become prominent. Over the years, I've drifted to the biological side of bioinformatics methods. The papers I write are similar to CS papers, but the projects are bigger, and I no longer submit papers to CS conferences. My typical experience with peer review is accept after 1-3 rounds of revisions. Hence I get to see more of the positive side, where the reviewers act as editors and their suggestions improve the paper. Of course, it's possible to get a CS-like experience with journals by being ambitious. If you try your luck by submitting the paper to prestigious journals it's unlikely to get in, your typical outcome is a desk rejection or rejection after a single round of reviews. Then you get to see more of the negative side of peer review. But I'm just not interested in playing that game.

sn41 4y ago
This is a very thoughtful take on the issue. My personal experience corroborates with this - CS conferences have more of these culture issues than mathematics journals, where the reviews tend to more thoughtful, and at the same time, considerate.

Archelaos 4y ago

> Instead, in CS, there is a tendency to hide behind an abrasive negative review when the reality is that the reviewer does not understand the paper. This problem goes even deeper, because it not only applies to papers but also to research grant applications. I even had once got a (positive) recommendation for one of my applications, but I could see from the statement that the reviewer did not really understand what was important.

bumby 4y ago
Not meant in a snarky way, but that almost feels to me as if the process worked. If they application committee did not see the value in the proposed work, it indicates they probably don't have much need for it. If they don't have much need, they shouldn't fund it. That's not to say the proposal isn't worthwhile, just that it didn't find the right audience. A contractor may have a whiz-bang new method of snow removal, but it doesn't mean people in Phoenix will want it (or even understand it).

hooby 4y ago

Just one question about that - what you are describing here - is that: .) an unsolvable problem inherent to all peer reviews? .) a problem that comes from specific practices, and could be solved by doing peer reviews differently? I'm just a layman - but if reviews are not done by peers (= experts working in the same field)... who else could properly review a paper?

jraph 4y ago
Your parent is not arguing against peer reviews, but against blind and double blind reviews. They are actually proposing that peer reviews are credited: > Instead, authored and credited reviews might lead to more responsible reviews, or reviewers respectfully declining when they might not know the topic. And yes, I do think reviewers should be credited. Because it's work, for one, that needed to be done so the paper could be published, and because it would probably indeed incentivize reviewers to do the work correctly. However, it could incentivize them to reject more papers, in fear of accepting a bad paper and be publicly advertised as a reviewer who reviewed and accepted it.