5 ms·

At the risk of sounding like a Reddit comment, what's the difference? I've had reviews on papers of people that didn't even skim the content. I've had rejection

At the risk of sounding like a Reddit comment, what's the difference? I've had reviews on papers of people that didn't even skim the content. I've had rejections from people who barely knew anything about the topic. I've had reviewers insist we should cite certain papers (funnily enough, all from the same lab even though they weren't really relevant). The peer review process is absolutely broken. There are no incentives to do good peer review, there are perverse incentives to stop "competing" papers from being published or modified with convenient citations. Adding ChatGPT to this process doesn't really change anything. It could even improve it: it'd be a crappy reviewer but at least you can test yourself what it's going to say so you can avoid the 6-12 months waiting period for a review.

bowsamic 4y ago
Yeah I am literally about to review a paper which I definitely don't understand well enough to review. I will probably just write something very vapid and brief. Half tempted to use GPT, but I'm a good enough writer that I can make it up.
- amelius 4y ago
  As opposed to being honest and trying to understand the paper, and if you fail then writing in your review that it is too difficult to understand?
  - bowsamic 4y ago
    Unfortunately both my mental health conditions and time constraints are such that I don't have the ability to spend time to understand the paper right now
    - janeway 4y ago
      Then don’t review it. I spend months or years writing papers. A voluntary reviewer who won’t bother reading is not helpful.
      - bowsamic 4y ago
        But then the journal may flag me as being unreliable. No thanks... I think I will just put in the shitty review
        tokai 4y ago
        But you are unreliable. Selfish people like you makes the world a worse place.
        bowsamic 4y ago
        Sorry but my career is an astronomically higher priority than any ideology of good peer review practises, and also an astronomically higher priority than "making the world a better place"
        tokai 4y ago
        Oh now I remember your post about your depressing post doc. Why do you care about getting marked as an unreliable reviewer if you are not staying in academia after the post doc? If its for your cv you can just write reviewer for what ever journal you want. Nobody is gonna check it.
        heisenbit 4y ago
        In Germany recently a medical specialist was jailed for 7.5 years for delivering wrong all clear results when reviewing skin cancer samples. He was aware he was not mentally healthy and was mot able to do his job. Please get help instead of playing pretend and barreling through. People will get hurt by wrong expert statements. You fear for career while at the same time putting the career of another at risk. People get respect for saying ‚no‘.
        bowsamic 4y ago
        I'm a quantum physicist, not a medical doctor.
    - titzer 4y ago
      You have to learn to say no to reviewing tasks that you can't do.
      - bowsamic 4y ago
        That's not how having a job works
        titzer 4y ago
        I get that you're stressed out and overworked, and I don't mean to add to it, but you seem to be confused between having an academic career and killing yourself. You're doing authors, editors, and yourself no favors by taking on load that you can't handle.
- tokai 4y ago
  Have thought about stopping being a reviewer?
  - bowsamic 4y ago
    Reviewing isn't an occupation. I just get endless emails badgering me into accepting to review papers. If you say you can't, they demand a suggestion for someone who knows more. The problem is, there aren't many people
    - tokai 4y ago
      It goes without saying that it isn't an occupation. You are not helping anyone by doing sub low quality reviewing. It would be better if noone reviewed than getting low effort reviews like what you are describing doing.
    - A_Venom_Roll 4y ago
      Unless it's specifically part of your jobto do so, you have the right to say 'no' to these emails, especially when you're already struggling with your mental health. Prioritize your own wellbeing over people who are so entitled that they make demands that are uncalled for.
derbOac 4y ago
I'm an editor at two journals and have been guest editor at others. I've reviewed countless papers. I think your observation isn't too far off actually. As a reviewer once I saw another reviewer (in the editor's decision letter) give as citations some papers that I was supposedly a co-author on. The problem is, those papers never existed; they were some weird amalgamation of citations from actual papers with my coauthors. They were exactly the sort of thing you'd expect from generative AI today, but at that time (to my knowledge) it didn't exist. I always assumed the reviewer was tired and miscopied something, which they probably did, but as you point out, it's hard to tell sometimes and sometimes it almost doesn't matter given the other things that go on. The thing that you're pointing out is that this sort stuff still goes on, and has been going on, even without ChatGPT. In this weird kind of meta-process, it's as if using ChatGPT to mimic citations is itself doing the sort of thing ChatGPT itself does. The thing no one seems to be mentioning is that the reviewers don't make the decisions, the editor does. The reviewers are supposed to be advisors or juries or something like that. There's an important discussion to be had about editorial quality at many journals, and I agree that this ChatGPT episode is just one illustration of the myriad ways academics is fundamentally broken, but it's an important thing to keep in mind.

bowsamic 4y ago

Yeah I am literally about to review a paper which I definitely don't understand well enough to review. I will probably just write something very vapid and brief. Half tempted to use GPT, but I'm a good enough writer that I can make it up.

amelius 4y ago
As opposed to being honest and trying to understand the paper, and if you fail then writing in your review that it is too difficult to understand?
- bowsamic 4y ago
  Unfortunately both my mental health conditions and time constraints are such that I don't have the ability to spend time to understand the paper right now
  - janeway 4y ago
    Then don’t review it. I spend months or years writing papers. A voluntary reviewer who won’t bother reading is not helpful.
    - bowsamic 4y ago
      But then the journal may flag me as being unreliable. No thanks... I think I will just put in the shitty review
      - tokai 4y ago
        But you are unreliable. Selfish people like you makes the world a worse place.
        bowsamic 4y ago
        Sorry but my career is an astronomically higher priority than any ideology of good peer review practises, and also an astronomically higher priority than "making the world a better place"
        tokai 4y ago
        Oh now I remember your post about your depressing post doc. Why do you care about getting marked as an unreliable reviewer if you are not staying in academia after the post doc? If its for your cv you can just write reviewer for what ever journal you want. Nobody is gonna check it.
      - heisenbit 4y ago
        In Germany recently a medical specialist was jailed for 7.5 years for delivering wrong all clear results when reviewing skin cancer samples. He was aware he was not mentally healthy and was mot able to do his job. Please get help instead of playing pretend and barreling through. People will get hurt by wrong expert statements. You fear for career while at the same time putting the career of another at risk. People get respect for saying ‚no‘.
        bowsamic 4y ago
        I'm a quantum physicist, not a medical doctor.
  - titzer 4y ago
    You have to learn to say no to reviewing tasks that you can't do.
    - bowsamic 4y ago
      That's not how having a job works
      - titzer 4y ago
        I get that you're stressed out and overworked, and I don't mean to add to it, but you seem to be confused between having an academic career and killing yourself. You're doing authors, editors, and yourself no favors by taking on load that you can't handle.
tokai 4y ago
Have thought about stopping being a reviewer?
- bowsamic 4y ago
  Reviewing isn't an occupation. I just get endless emails badgering me into accepting to review papers. If you say you can't, they demand a suggestion for someone who knows more. The problem is, there aren't many people
  - tokai 4y ago
    It goes without saying that it isn't an occupation. You are not helping anyone by doing sub low quality reviewing. It would be better if noone reviewed than getting low effort reviews like what you are describing doing.
  - A_Venom_Roll 4y ago
    Unless it's specifically part of your jobto do so, you have the right to say 'no' to these emails, especially when you're already struggling with your mental health. Prioritize your own wellbeing over people who are so entitled that they make demands that are uncalled for.

derbOac 4y ago

I'm an editor at two journals and have been guest editor at others. I've reviewed countless papers. I think your observation isn't too far off actually. As a reviewer once I saw another reviewer (in the editor's decision letter) give as citations some papers that I was supposedly a co-author on. The problem is, those papers never existed; they were some weird amalgamation of citations from actual papers with my coauthors. They were exactly the sort of thing you'd expect from generative AI today, but at that time (to my knowledge) it didn't exist. I always assumed the reviewer was tired and miscopied something, which they probably did, but as you point out, it's hard to tell sometimes and sometimes it almost doesn't matter given the other things that go on. The thing that you're pointing out is that this sort stuff still goes on, and has been going on, even without ChatGPT. In this weird kind of meta-process, it's as if using ChatGPT to mimic citations is itself doing the sort of thing ChatGPT itself does. The thing no one seems to be mentioning is that the reviewers don't make the decisions, the editor does. The reviewers are supposed to be advisors or juries or something like that. There's an important discussion to be had about editorial quality at many journals, and I agree that this ChatGPT episode is just one illustration of the myriad ways academics is fundamentally broken, but it's an important thing to keep in mind.