8 ms·

“They also tend to over-estimate the effect of peer review (often equating peer review with validity).“ In my experience, scientists ate comfortably cynical ab

by vibrio 3y ago

“They also tend to over-estimate the effect of peer review (often equating peer review with validity).“ In my experience, scientists ate comfortably cynical about peer review- even those that serve as reviewers and editors- except maybe junior scientists that haven’t gotten burned yet.

kergonath 3y ago
Yes, because we know how the metaphorical sausage is made: with unpaid reviewers who have many other, more interesting things to do and often an axe to grind. That is, if they don’t delegate the review to one of their post-docs.
- aftoprokrustes 3y ago
  Post doc? In what kind of utopian field did you work? In my former institute virtually all papers were written by PhD candidates, and reviewed by PhD candidates. With the expected effect on quality (due to lack of experience and impostor-syndrome-induced "how can I propose to reject? They are likely better than me"). But the Prof-to-postdoc-to-PhD-ratio was particularly bad (1-2-15).
  - kelipso 3y ago
    I was reviewing papers starting second semester of grad school with my advisor just signing off on it, so not even PhD candidates, and it was the same for my lab mates too. Initially we spent probably a few hours on a paper for peer review because we were relatively unfamiliar with the field but eventually I spent maybe a couple of hours doing the review. Wouldn't say peer review is a joke but it's definitely overrated by the public.
  - godelski 3y ago
    > "how can I propose to reject? They are likely better than me" Funny enough, I see exactly the opposite. I've seen this in both reviews I've done and reviews I've received. Just this week I reviewed and saw one of my fellow reviewers write in their justifications: I am not familiar with X, but I am skeptical that the method can scale to a more complex application. Their weaknesses section was extremely generic and it was very clear they didn't understand the work. They gave a weak reject. In fact, when I first started reviewing, I was explicitly told to _only_ accept if I was confident that the work was good. So in my experience, the bias goes the other way that you are proposing. Btw, I've even seen undergrads acting as reviewers. I was asked to review in in my first year of grad school. I don't think I was qualified then, but I was always a junior reviewer rather than a full so idk.
- godelski 3y ago
  > with unpaid reviewers who have many other, more interesting things to do It's kinda funny. A journal doesn't make the product it sells (the papers that it copywrites). It doesn't pay for the service it performs ("vetting" and editing). And both of these would be done regardless of their existence. I can understand distribution, but that hasn't been useful for over a decade now. What even do these things do anymore? (btw, I've seen profs delegate to undergrads. And it is quite common for post-docs AND grad students to be reviewers. Trust me, I am one)
  - kergonath 3y ago
    > What even do these things do anymore? Networking, mostly, in the sense that an article in a high impact journal has a higher probability to be integrated in citations networks. The fact that there is some gate keeping means that it’s valuable to be in rather than out, and that’s something you can use to get a position. Also, better journals (which are not necessarily the highest-impact ones) tend to have more thorough peer review (such as 3 reviewers by default instead of 1 or 2, editors who are not afraid to ask for more reviews if the 3 are not conclusive, etc). > (btw, I've seen profs delegate to undergrads. And it is quite common for post-docs AND grad students to be reviewers. Trust me, I am one) I am lucky not to have been in that situation when I was a student, and I did not delegate any further when I got the occasional review from the prof when I was a post-doc. But I am unfortunately not surprised.
    - godelski 3y ago
      I am not asking "what do they do" in the sense of bureaucratic circular logic, but rather what do they serve us, as scientists, that is a unique thing that is not solved by other things that we do. What field are you in where post-docs aren't getting calls to review directly from the conference? I'm in ML and it isn't my advisor assigning me reviews, it is the conference.
jakear 3y ago
It's the general public that equates "peer reviewed" with "definitely correct, does not need to be questioned".
- PaulHoule 3y ago
  Except for the ones that conclude it is all a scam.
- kibibu 3y ago
  There is genuine merit in peer review though. A lot of the marketing "papers" from OpenAI and Google would benefit from going through a peer review process instead of scholarwashing and uploading straight to Arxiv. One simple example, (from memory) the Bard paper doesn't include results for experiments in which GPT-4 outperforms it. As a result, people come away from these works with an inflated understanding of their capabilities. This wouldn't pass peer review.
  - jakear 3y ago
    Industry doesn't need peer review. The proof of the pudding is in the sales.
    - ModernMech 3y ago
      This is the final stage of capitalism, where market success is conflated with scientific rigor.
      - jakear 3y ago
        Engineering, not scientific. High Sales = A large number of people can attest to the engineered good or service being of high enough quality that they will exchange hard earned money for the ability to use it. Peer Review = Some folks who derive self-worth from citations ask you to add a citation to their work and you do it because you'll probably need to ask them for the same some time later.
renonce 3y ago
I don't know how scientists handle peer review but aren't they fighting with peer review to get their papers published and apply for PhD and tenure and grants etc with these publications?
godelski 3y ago
It's because we all know it is a con to just get money from universities and governments. We had a long time without this system and we perverted it from its original intent. 3-4 people reading a paper are incapable of proving something is valid (the inverse is true, but also remember there's 3 options: true, false, indeterminate). It's even more silly that we think we can rank them (i.e. top tier venues). Rejection rate isn't a ranking, and it is silly to think it could be (way too easy to hack). We just go with the momentum and because that's what the admins require of us. Otherwise, let's me honest, we're all communicating our work outside the structure of journals and conferences (at least they offer meetups). It's very hard for me to justify their existence if we're being honest. And honestly, I don't see how abandoning these systems wouldn't just save us time and money. Like god, the number of hours I spend reworking things which got no actionable feedback from the previous round, is insane. It is just too easy to get non-actionable rejections and that just wastes everybody's time and money, and frankly holds back science.

by vibrio 3y ago

kergonath 3y ago

Yes, because we know how the metaphorical sausage is made: with unpaid reviewers who have many other, more interesting things to do and often an axe to grind. That is, if they don’t delegate the review to one of their post-docs.

aftoprokrustes 3y ago
Post doc? In what kind of utopian field did you work? In my former institute virtually all papers were written by PhD candidates, and reviewed by PhD candidates. With the expected effect on quality (due to lack of experience and impostor-syndrome-induced "how can I propose to reject? They are likely better than me"). But the Prof-to-postdoc-to-PhD-ratio was particularly bad (1-2-15).
- kelipso 3y ago
  I was reviewing papers starting second semester of grad school with my advisor just signing off on it, so not even PhD candidates, and it was the same for my lab mates too. Initially we spent probably a few hours on a paper for peer review because we were relatively unfamiliar with the field but eventually I spent maybe a couple of hours doing the review. Wouldn't say peer review is a joke but it's definitely overrated by the public.
- godelski 3y ago
  > "how can I propose to reject? They are likely better than me" Funny enough, I see exactly the opposite. I've seen this in both reviews I've done and reviews I've received. Just this week I reviewed and saw one of my fellow reviewers write in their justifications: I am not familiar with X, but I am skeptical that the method can scale to a more complex application. Their weaknesses section was extremely generic and it was very clear they didn't understand the work. They gave a weak reject. In fact, when I first started reviewing, I was explicitly told to _only_ accept if I was confident that the work was good. So in my experience, the bias goes the other way that you are proposing. Btw, I've even seen undergrads acting as reviewers. I was asked to review in in my first year of grad school. I don't think I was qualified then, but I was always a junior reviewer rather than a full so idk.
godelski 3y ago
> with unpaid reviewers who have many other, more interesting things to do It's kinda funny. A journal doesn't make the product it sells (the papers that it copywrites). It doesn't pay for the service it performs ("vetting" and editing). And both of these would be done regardless of their existence. I can understand distribution, but that hasn't been useful for over a decade now. What even do these things do anymore? (btw, I've seen profs delegate to undergrads. And it is quite common for post-docs AND grad students to be reviewers. Trust me, I am one)
- kergonath 3y ago
  > What even do these things do anymore? Networking, mostly, in the sense that an article in a high impact journal has a higher probability to be integrated in citations networks. The fact that there is some gate keeping means that it’s valuable to be in rather than out, and that’s something you can use to get a position. Also, better journals (which are not necessarily the highest-impact ones) tend to have more thorough peer review (such as 3 reviewers by default instead of 1 or 2, editors who are not afraid to ask for more reviews if the 3 are not conclusive, etc). > (btw, I've seen profs delegate to undergrads. And it is quite common for post-docs AND grad students to be reviewers. Trust me, I am one) I am lucky not to have been in that situation when I was a student, and I did not delegate any further when I got the occasional review from the prof when I was a post-doc. But I am unfortunately not surprised.
  - godelski 3y ago
    I am not asking "what do they do" in the sense of bureaucratic circular logic, but rather what do they serve us, as scientists, that is a unique thing that is not solved by other things that we do. What field are you in where post-docs aren't getting calls to review directly from the conference? I'm in ML and it isn't my advisor assigning me reviews, it is the conference.

jakear 3y ago

It's the general public that equates "peer reviewed" with "definitely correct, does not need to be questioned".

PaulHoule 3y ago
Except for the ones that conclude it is all a scam.
kibibu 3y ago
There is genuine merit in peer review though. A lot of the marketing "papers" from OpenAI and Google would benefit from going through a peer review process instead of scholarwashing and uploading straight to Arxiv. One simple example, (from memory) the Bard paper doesn't include results for experiments in which GPT-4 outperforms it. As a result, people come away from these works with an inflated understanding of their capabilities. This wouldn't pass peer review.
- jakear 3y ago
  Industry doesn't need peer review. The proof of the pudding is in the sales.
  - ModernMech 3y ago
    This is the final stage of capitalism, where market success is conflated with scientific rigor.
    - jakear 3y ago
      Engineering, not scientific. High Sales = A large number of people can attest to the engineered good or service being of high enough quality that they will exchange hard earned money for the ability to use it. Peer Review = Some folks who derive self-worth from citations ask you to add a citation to their work and you do it because you'll probably need to ask them for the same some time later.

renonce 3y ago

I don't know how scientists handle peer review but aren't they fighting with peer review to get their papers published and apply for PhD and tenure and grants etc with these publications?

godelski 3y ago

It's because we all know it is a con to just get money from universities and governments. We had a long time without this system and we perverted it from its original intent. 3-4 people reading a paper are incapable of proving something is valid (the inverse is true, but also remember there's 3 options: true, false, indeterminate). It's even more silly that we think we can rank them (i.e. top tier venues). Rejection rate isn't a ranking, and it is silly to think it could be (way too easy to hack). We just go with the momentum and because that's what the admins require of us. Otherwise, let's me honest, we're all communicating our work outside the structure of journals and conferences (at least they offer meetups). It's very hard for me to justify their existence if we're being honest. And honestly, I don't see how abandoning these systems wouldn't just save us time and money. Like god, the number of hours I spend reworking things which got no actionable feedback from the previous round, is insane. It is just too easy to get non-actionable rejections and that just wastes everybody's time and money, and frankly holds back science.