8 ms·

Great example of why the peer review system is broke. I've seen first hand how the peer review system works in the United States. Foreign Postdocs studying in

Great example of why the peer review system is broke. I've seen first hand how the peer review system works in the United States. Foreign Postdocs studying in the United States are desperate for a Green Card. They peer review as many papers as they can as quickly as they can to prove something or other for the purposes of a securing a Green Card. Some of these foreign postdocs do a good job, many do not. Also, overworked professors don't always take the time to carefully read a paper and don't care. The result is that a lot papers make it through peer review that shouldn't. I'd rather have my paper peer reviewed live on the internet by allowing people to comment on it beneath a PDF link than to suffer through another drawn out peer review process. People who care have a text-box where they say what they want, and if no one cares then it indicates I need to do research on something else!

leemailll 9y ago
Seriously, I don't know any top quality journals is using large quantity of postdocs as reviewers. In academy if one is invited by a top journal as a reviewer, that is an honor, meaning one is acknowledged by peers. In my field I can't think of any quality journals' reviewers are majorly post docs. Also your pointing fingers at foreigners is quite disturbing. Why you think they caused the problem? Do you have any evidence? I don't think in any academic workplace you can say this without unfavorable consequence.
- FiReaNG3L 9y ago
  What does happen very frequently however is a prestigious researcher being asked to review papers, and passing them to his/her postdocs.
- zionic 9y ago
  He mentioned them because their unique situation creates perverse incentives. Your last line is one of the most frustrating parts of academia. Not being able to criticize misaligned incentives simply because it involves foreigners is crazy.
  - leemailll 9y ago
    >Your last line is one of the most frustrating parts of academia. Not being able to criticize misaligned incentives simply because it involves foreigners is crazy. I don't know why you summarize my lines this way. I basically want to know whether he can offer some evidence to support that idea.
77pt77 9y ago
Is a green card that much of a big deal to justify that kind of behavior?
cing 9y ago
The peer review system is broke because this manuscript was published? You seem to be arguing two opposing ideas. More rigorous peer review, be that on a pre-print server (comment box) or during publication (suffering the drawn out process), would've prevented this paper from being published. Many journals welcome post-publication review, like this response, as well. There are reasons why the peer review system is broken, but this doesn't seem like one of them.
a_bonobo 9y ago
>Foreign Postdocs studying in the United States are desperate for a Green Card. They peer review as many papers as they can as quickly as they can to prove something or other for the purposes of a securing a Green Card. Some of these foreign postdocs do a good job, many do not. A I have never heard of this. No-one knows if you review a paper, it has no bearing on your job security or your visa status, this post is baffling
- dsfyu404ed 9y ago
  You might not be credited by name but having a college professor who is willing to say you are indispensable to the productivity of the department isn't worth nothing.
  - ptero 9y ago
    He would likely be more inclined to argue this if a postdoc helps his lab produce more research or takes care of a lot of admin stuff
    - a_bonobo 9y ago
      That's it - a publication from the postdoc is, in career terms, worth a fortune, while a paper review is worth a nickel
- pcrh 9y ago
  The comment on visas potentially refers to the fact that one of the criteria for certain visa categories is that one be distinguished in one's field. Serving as a reviewer demonstrates that one has acted as a judge of one's peers, and is therefore of superior standing.
zzz95 9y ago
I do not think this is correct, at least for most of the sub-fields of computer science and physics. Prestigious conferences and journals invite only well known researchers to review, who are mostly professors. Few post-docs get invited. Each paper gets multiple reviews (3-5), and conflicting reviews get discussed making the review process quite rigorous. Reviewing a paper is mostly charity and gets you nothing in return, except the fact that when you submit a paper, someone will do the same. It does not help in the Green Card process at all, or boosts your academic credentials. Regarding overworked professors, they are allowed to delegate reviews internally to students to ease the time they have to spend.
- wfunction 9y ago
  > I do not think this is correct, at least for most of the sub-fields of computer science and physics. Prestigious conferences and journals invite only well known researchers to review, who are mostly professors. Few post-docs get invited. What about PhDs? You've never seen PhDs review papers in CS? It happens for prestigious conferences too.
  - seanmcdirmid 9y ago
    CS is diverse in field cultures but ya. You don't even need a PhD to be on a PC, but usually every field has a crowd of reviewers that defines it by constantly being on PCs.
- 88e282102ae2e5b 9y ago
  It seems that for the glamor journals in biology it definitely is broken, at least for click-baity manuscripts like this. This paper was immediately shat upon by everyone in my department. There's a running joke often said when things like this crop up: 'just because it's in Cell/Nature/Science doesn't mean it's false".
- jacquesm 9y ago
  Experience in CS, Math or Physics does not necessarily carry over to other branches of science.
bmsran 9y ago
Also a great example of the system working as intended. A group publishes a dubious study with potentially important implications and people queue up to refute it, correcting the scientific record.
language_ninja 9y ago
broke: having completely run out of money [1] broken: having been fractured or damaged and no longer in one piece or in working order [2] 1. http://lmgtfy.com/?q=define%3A+broke http://lmgtfy.com/?q=define%3A+broke 2. http://lmgtfy.com/?q=define%3A+broken http://lmgtfy.com/?q=define%3A+broken
eternalban 9y ago
> Great example of why the peer review system is broke. Don't know about that, but the inbred cloned mouse business seems to have quality control issues.
titzer 9y ago
> Great example of why the peer review system is broke. s/broke/broken/ (unless you meant bankrupt, then broke is fine). > Foreign Postdocs studying in the United States are desperate for a Green Card. They peer review as many papers as they can as quickly as they can to prove something or other for the purposes of a securing a Green Card. Some of these foreign postdocs do a good job, many do not. Also, overworked professors don't always take the time to carefully read a paper and don't care. The result is that a lot papers make it through peer review that shouldn't. Well, it's unfortunate that you seem to have experienced the worst review processes. It is also unfortunate that you automatically assume that it's _foreign_ postdocs that are doing the shoddy work. Maybe you had bad luck, or maybe you're oversimplifying for the sake of argument, dunno, but that does not jive very well with what I've seen, all the way up from being a student author, student reviewer, reviewer, PC member, journal reviewer, and Program Chair of a conference. At least in CS, the number of reviews done by postdocs is not a large percentage. Most PCs are composed of academic and industry researchers, primarily professors and professionals, and increasingly reviewers are no longer allowed to use sub-reviewers without attribution. The number of postdoc PC members is low; and for good reason. Postdocs typically last less than two years, and during this period they are more focused on publishing their own research as a last push to improve their academic standing before finding a job. I get the impression that it's known that postdocs are too busy to do a good job at reviewing. > I'd rather have my paper peer reviewed live on the internet by allowing people to comment on it beneath a PDF link... I guess you never had a Boaty McBoatface moment. I think open review without experts would be disastrous, as the nitty gritty fine details that absolutely have to be right should be vetted carefully by experts, rather than spending forever in a shouting match with the loudest voices in the crowd. Can you imagine peer review by YouTube comments? shudder

leemailll 9y ago

Seriously, I don't know any top quality journals is using large quantity of postdocs as reviewers. In academy if one is invited by a top journal as a reviewer, that is an honor, meaning one is acknowledged by peers. In my field I can't think of any quality journals' reviewers are majorly post docs. Also your pointing fingers at foreigners is quite disturbing. Why you think they caused the problem? Do you have any evidence? I don't think in any academic workplace you can say this without unfavorable consequence.

FiReaNG3L 9y ago
What does happen very frequently however is a prestigious researcher being asked to review papers, and passing them to his/her postdocs.
zionic 9y ago
He mentioned them because their unique situation creates perverse incentives. Your last line is one of the most frustrating parts of academia. Not being able to criticize misaligned incentives simply because it involves foreigners is crazy.
- leemailll 9y ago
  >Your last line is one of the most frustrating parts of academia. Not being able to criticize misaligned incentives simply because it involves foreigners is crazy. I don't know why you summarize my lines this way. I basically want to know whether he can offer some evidence to support that idea.

77pt77 9y ago

Is a green card that much of a big deal to justify that kind of behavior?

cing 9y ago

The peer review system is broke because this manuscript was published? You seem to be arguing two opposing ideas. More rigorous peer review, be that on a pre-print server (comment box) or during publication (suffering the drawn out process), would've prevented this paper from being published. Many journals welcome post-publication review, like this response, as well. There are reasons why the peer review system is broken, but this doesn't seem like one of them.

a_bonobo 9y ago

>Foreign Postdocs studying in the United States are desperate for a Green Card. They peer review as many papers as they can as quickly as they can to prove something or other for the purposes of a securing a Green Card. Some of these foreign postdocs do a good job, many do not. A I have never heard of this. No-one knows if you review a paper, it has no bearing on your job security or your visa status, this post is baffling

dsfyu404ed 9y ago
You might not be credited by name but having a college professor who is willing to say you are indispensable to the productivity of the department isn't worth nothing.
- ptero 9y ago
  He would likely be more inclined to argue this if a postdoc helps his lab produce more research or takes care of a lot of admin stuff
  - a_bonobo 9y ago
    That's it - a publication from the postdoc is, in career terms, worth a fortune, while a paper review is worth a nickel
pcrh 9y ago
The comment on visas potentially refers to the fact that one of the criteria for certain visa categories is that one be distinguished in one's field. Serving as a reviewer demonstrates that one has acted as a judge of one's peers, and is therefore of superior standing.

zzz95 9y ago

I do not think this is correct, at least for most of the sub-fields of computer science and physics. Prestigious conferences and journals invite only well known researchers to review, who are mostly professors. Few post-docs get invited. Each paper gets multiple reviews (3-5), and conflicting reviews get discussed making the review process quite rigorous. Reviewing a paper is mostly charity and gets you nothing in return, except the fact that when you submit a paper, someone will do the same. It does not help in the Green Card process at all, or boosts your academic credentials. Regarding overworked professors, they are allowed to delegate reviews internally to students to ease the time they have to spend.

wfunction 9y ago
> I do not think this is correct, at least for most of the sub-fields of computer science and physics. Prestigious conferences and journals invite only well known researchers to review, who are mostly professors. Few post-docs get invited. What about PhDs? You've never seen PhDs review papers in CS? It happens for prestigious conferences too.
- seanmcdirmid 9y ago
  CS is diverse in field cultures but ya. You don't even need a PhD to be on a PC, but usually every field has a crowd of reviewers that defines it by constantly being on PCs.
88e282102ae2e5b 9y ago
It seems that for the glamor journals in biology it definitely is broken, at least for click-baity manuscripts like this. This paper was immediately shat upon by everyone in my department. There's a running joke often said when things like this crop up: 'just because it's in Cell/Nature/Science doesn't mean it's false".
jacquesm 9y ago
Experience in CS, Math or Physics does not necessarily carry over to other branches of science.

bmsran 9y ago

Also a great example of the system working as intended. A group publishes a dubious study with potentially important implications and people queue up to refute it, correcting the scientific record.

language_ninja 9y ago

broke: having completely run out of money [1] broken: having been fractured or damaged and no longer in one piece or in working order [2] 1. http://lmgtfy.com/?q=define%3A+broke http://lmgtfy.com/?q=define%3A+broke 2. http://lmgtfy.com/?q=define%3A+broken http://lmgtfy.com/?q=define%3A+broken

eternalban 9y ago

> Great example of why the peer review system is broke. Don't know about that, but the inbred cloned mouse business seems to have quality control issues.

titzer 9y ago

> Great example of why the peer review system is broke. s/broke/broken/ (unless you meant bankrupt, then broke is fine). > Foreign Postdocs studying in the United States are desperate for a Green Card. They peer review as many papers as they can as quickly as they can to prove something or other for the purposes of a securing a Green Card. Some of these foreign postdocs do a good job, many do not. Also, overworked professors don't always take the time to carefully read a paper and don't care. The result is that a lot papers make it through peer review that shouldn't. Well, it's unfortunate that you seem to have experienced the worst review processes. It is also unfortunate that you automatically assume that it's _foreign_ postdocs that are doing the shoddy work. Maybe you had bad luck, or maybe you're oversimplifying for the sake of argument, dunno, but that does not jive very well with what I've seen, all the way up from being a student author, student reviewer, reviewer, PC member, journal reviewer, and Program Chair of a conference. At least in CS, the number of reviews done by postdocs is not a large percentage. Most PCs are composed of academic and industry researchers, primarily professors and professionals, and increasingly reviewers are no longer allowed to use sub-reviewers without attribution. The number of postdoc PC members is low; and for good reason. Postdocs typically last less than two years, and during this period they are more focused on publishing their own research as a last push to improve their academic standing before finding a job. I get the impression that it's known that postdocs are too busy to do a good job at reviewing. > I'd rather have my paper peer reviewed live on the internet by allowing people to comment on it beneath a PDF link... I guess you never had a Boaty McBoatface moment. I think open review without experts would be disastrous, as the nitty gritty fine details that absolutely have to be right should be vetted carefully by experts, rather than spending forever in a shouting match with the loudest voices in the crowd. Can you imagine peer review by YouTube comments? shudder