3 ms·

Yes, because we know how the metaphorical sausage is made: with unpaid reviewers who have many other, more interesting things to do and often an axe to grind. T

Yes, because we know how the metaphorical sausage is made: with unpaid reviewers who have many other, more interesting things to do and often an axe to grind. That is, if they don’t delegate the review to one of their post-docs.

aftoprokrustes 3y ago
Post doc? In what kind of utopian field did you work? In my former institute virtually all papers were written by PhD candidates, and reviewed by PhD candidates. With the expected effect on quality (due to lack of experience and impostor-syndrome-induced "how can I propose to reject? They are likely better than me"). But the Prof-to-postdoc-to-PhD-ratio was particularly bad (1-2-15).
- kelipso 3y ago
  I was reviewing papers starting second semester of grad school with my advisor just signing off on it, so not even PhD candidates, and it was the same for my lab mates too. Initially we spent probably a few hours on a paper for peer review because we were relatively unfamiliar with the field but eventually I spent maybe a couple of hours doing the review. Wouldn't say peer review is a joke but it's definitely overrated by the public.
- godelski 3y ago
  > "how can I propose to reject? They are likely better than me" Funny enough, I see exactly the opposite. I've seen this in both reviews I've done and reviews I've received. Just this week I reviewed and saw one of my fellow reviewers write in their justifications: I am not familiar with X, but I am skeptical that the method can scale to a more complex application. Their weaknesses section was extremely generic and it was very clear they didn't understand the work. They gave a weak reject. In fact, when I first started reviewing, I was explicitly told to _only_ accept if I was confident that the work was good. So in my experience, the bias goes the other way that you are proposing. Btw, I've even seen undergrads acting as reviewers. I was asked to review in in my first year of grad school. I don't think I was qualified then, but I was always a junior reviewer rather than a full so idk.
godelski 3y ago
> with unpaid reviewers who have many other, more interesting things to do It's kinda funny. A journal doesn't make the product it sells (the papers that it copywrites). It doesn't pay for the service it performs ("vetting" and editing). And both of these would be done regardless of their existence. I can understand distribution, but that hasn't been useful for over a decade now. What even do these things do anymore? (btw, I've seen profs delegate to undergrads. And it is quite common for post-docs AND grad students to be reviewers. Trust me, I am one)
- kergonath 3y ago
  > What even do these things do anymore? Networking, mostly, in the sense that an article in a high impact journal has a higher probability to be integrated in citations networks. The fact that there is some gate keeping means that it’s valuable to be in rather than out, and that’s something you can use to get a position. Also, better journals (which are not necessarily the highest-impact ones) tend to have more thorough peer review (such as 3 reviewers by default instead of 1 or 2, editors who are not afraid to ask for more reviews if the 3 are not conclusive, etc). > (btw, I've seen profs delegate to undergrads. And it is quite common for post-docs AND grad students to be reviewers. Trust me, I am one) I am lucky not to have been in that situation when I was a student, and I did not delegate any further when I got the occasional review from the prof when I was a post-doc. But I am unfortunately not surprised.
  - godelski 3y ago
    I am not asking "what do they do" in the sense of bureaucratic circular logic, but rather what do they serve us, as scientists, that is a unique thing that is not solved by other things that we do. What field are you in where post-docs aren't getting calls to review directly from the conference? I'm in ML and it isn't my advisor assigning me reviews, it is the conference.

aftoprokrustes 3y ago

Post doc? In what kind of utopian field did you work? In my former institute virtually all papers were written by PhD candidates, and reviewed by PhD candidates. With the expected effect on quality (due to lack of experience and impostor-syndrome-induced "how can I propose to reject? They are likely better than me"). But the Prof-to-postdoc-to-PhD-ratio was particularly bad (1-2-15).

kelipso 3y ago
I was reviewing papers starting second semester of grad school with my advisor just signing off on it, so not even PhD candidates, and it was the same for my lab mates too. Initially we spent probably a few hours on a paper for peer review because we were relatively unfamiliar with the field but eventually I spent maybe a couple of hours doing the review. Wouldn't say peer review is a joke but it's definitely overrated by the public.
godelski 3y ago
> "how can I propose to reject? They are likely better than me" Funny enough, I see exactly the opposite. I've seen this in both reviews I've done and reviews I've received. Just this week I reviewed and saw one of my fellow reviewers write in their justifications: I am not familiar with X, but I am skeptical that the method can scale to a more complex application. Their weaknesses section was extremely generic and it was very clear they didn't understand the work. They gave a weak reject. In fact, when I first started reviewing, I was explicitly told to _only_ accept if I was confident that the work was good. So in my experience, the bias goes the other way that you are proposing. Btw, I've even seen undergrads acting as reviewers. I was asked to review in in my first year of grad school. I don't think I was qualified then, but I was always a junior reviewer rather than a full so idk.

godelski 3y ago

> with unpaid reviewers who have many other, more interesting things to do It's kinda funny. A journal doesn't make the product it sells (the papers that it copywrites). It doesn't pay for the service it performs ("vetting" and editing). And both of these would be done regardless of their existence. I can understand distribution, but that hasn't been useful for over a decade now. What even do these things do anymore? (btw, I've seen profs delegate to undergrads. And it is quite common for post-docs AND grad students to be reviewers. Trust me, I am one)

kergonath 3y ago
> What even do these things do anymore? Networking, mostly, in the sense that an article in a high impact journal has a higher probability to be integrated in citations networks. The fact that there is some gate keeping means that it’s valuable to be in rather than out, and that’s something you can use to get a position. Also, better journals (which are not necessarily the highest-impact ones) tend to have more thorough peer review (such as 3 reviewers by default instead of 1 or 2, editors who are not afraid to ask for more reviews if the 3 are not conclusive, etc). > (btw, I've seen profs delegate to undergrads. And it is quite common for post-docs AND grad students to be reviewers. Trust me, I am one) I am lucky not to have been in that situation when I was a student, and I did not delegate any further when I got the occasional review from the prof when I was a post-doc. But I am unfortunately not surprised.
- godelski 3y ago
  I am not asking "what do they do" in the sense of bureaucratic circular logic, but rather what do they serve us, as scientists, that is a unique thing that is not solved by other things that we do. What field are you in where post-docs aren't getting calls to review directly from the conference? I'm in ML and it isn't my advisor assigning me reviews, it is the conference.