3 ms·

Meanwhile: > It has come to my attention that academics are now using generative AI (Chat GPT or whatever) to conduct their peer reviews. > [..] > If ever t

Meanwhile: > It has come to my attention that academics are now using generative AI (Chat GPT or whatever) to conduct their peer reviews. > [..] > If ever there were a compelling argument to totally abandon the intellectually dishonest notion of “blind” peer review – a system everyone knows is broken, rarely fully “blind” and frequently not “blind” at all, but allowing unscrupulous or mediocre yet established scholars to sabotage promising work – then this finally is it: accountability and transparency in the face of the robot takeover. Every person who reviews another person’s scholarship must be willing to sign their name to their own evaluation, to stand by it and assert it was not the work of the machines. Unethical academics, AI, and peer review https://nicospage.eu/blog https://nicospage.eu/blog

bee_rider 3y ago
It is far too early to say there’s some ethics convention for not using ML models to write peer reviews. If you used a machine to write your peer review, you’ve staked your reputation on its output being correct (in the same way that you stake your reputation on not producing bullshit peer reviews, which is to say… eh, probably not a make-or-break thing but it is in the mix). So you need to check the output. That’s the skill we as a society value. We don’t employ scientists for their literature skills, but for their ability to build and evaluate theories and data, that sort of thing. And hey, less brainpower remembering grammar rules means more for the study of science. This is good. Some people will torpedo their careers by putting too much faith in the ML model, but the way to avoid being them is to apply the same level of diligence to the model’s output as any other tool. I’m more worried that ML models will enable some continued silliness. There’s something odd going on if people type their actual arguments into an ML model, then it produces extra necessary filler text to get into a journal, then we use ML models to review that text and hope to distill it to the actual arguments, and then the general public again uses ML models to do the same. It seems like we’d just be better off sharing the prompts or something, hahaha.
- auggierose 3y ago
  You are not staking anything with your peer review, certainly not your reputation.
hashtag-til 3y ago
Soon: AI generated paper evaluated by AI reviewer - nobody learns anything.

bee_rider 3y ago

It is far too early to say there’s some ethics convention for not using ML models to write peer reviews. If you used a machine to write your peer review, you’ve staked your reputation on its output being correct (in the same way that you stake your reputation on not producing bullshit peer reviews, which is to say… eh, probably not a make-or-break thing but it is in the mix). So you need to check the output. That’s the skill we as a society value. We don’t employ scientists for their literature skills, but for their ability to build and evaluate theories and data, that sort of thing. And hey, less brainpower remembering grammar rules means more for the study of science. This is good. Some people will torpedo their careers by putting too much faith in the ML model, but the way to avoid being them is to apply the same level of diligence to the model’s output as any other tool. I’m more worried that ML models will enable some continued silliness. There’s something odd going on if people type their actual arguments into an ML model, then it produces extra necessary filler text to get into a journal, then we use ML models to review that text and hope to distill it to the actual arguments, and then the general public again uses ML models to do the same. It seems like we’d just be better off sharing the prompts or something, hahaha.

auggierose 3y ago
You are not staking anything with your peer review, certainly not your reputation.

hashtag-til 3y ago

Soon: AI generated paper evaluated by AI reviewer - nobody learns anything.