4 ms·

That's probably a good keeping-your-head-low strategy, but I can't help but feel this doesn't treat it as being severe as it is. I don't want academia to overre

That's probably a good keeping-your-head-low strategy, but I can't help but feel this doesn't treat it as being severe as it is. I don't want academia to overreact about LLMs (they have done this enough already, with the huge number of academic cheating accusations) but AI output that is entirely unchecked doesn't belong in the scientific peer review process. Those using AI tools in such situations should be expected to remove anything from the LLM's output that they can't verify with their own expertise. Reviewing out of your expertise doesn't necessarily inevitably lead to mistakes, but unchecked AI output will.

bell-cot 3y ago
In theory, I mostly agree with you. In practice - my advice is for the academic, who is trying to get an article published in "one of the well-reputable journals". That is a weak hand to be playing. Vs. the journal's editor is in a far stronger position, to hit back hard at whoever seems to be farming out their review job to a cut-rate bot. Edit: 's/is farming/seems to be farming/'
creer 3y ago
Yes, and the point then is that it is unchecked. Would it be different if it had been farmed out to a student - and then unchecked? Or for that matter if it had been dictated - and then unchecked?
- creatonez 3y ago
  That could rise to the level of serious infraction too, of course. But there is one key difference that makes it worse, which is that a student is not probabilistic like an LLM is. Given enough output, no matter how well behaved you think your prompt is, the LLM will output something completely off the rails. In fact, it will give every possible output. Can happen to students too (sleep deprivation will do it), but not with the same inevitability as the LLM tool. To phrase another way, you could set up the 'human factor' in a way that you can trust unchecked output from another human (e.g. checking their expertise in academia, or if they are a commercial aircraft pilot, checking their pre-flight notes on how much sleep they got), but not for LLMs.
skywhopper 3y ago
Agree that this is a potentially system-damaging problem that will only get worse if not directly dealt with. In this case, I think the advice in the OP is good however: address the feedback from the good review, resubmit, and once the paper is accepted, then contact the editor with concerns once it's clear that you aren't objecting to needing to revise.
SubiculumCode 3y ago
but again, there is no evidence of AI, just incompetence and/or laziness.
PeterisP 3y ago
It's about informing the only person who can get extra information, decide if it's severe and do something about it - the editor. The author can't (and shouldn't) do anything directly about the anonymous reviewer, all the responsibility, authority and duty is up to the editor, who at least knows who that person is.
- creatonez 3y ago
  Of course. I would do pretty much the same thing (finishing the review process first before complaining), but I would mention the suspicion of AI usage. If that aspect is missed, then it's very much possible the investigation won't lead to the necessary improvement in the organization. It's worth speaking out about such a problem if you are comfortable enough in your career to risk it, because it represents something that's going to rot the entire journal and the scientific community as a whole.
NoMoreNicksLeft 3y ago
Just out of curiosity, are you aware that some lawyers have started submitting LLM-generated content as legal drafts in (US) courts of law? Last I heard, the one had been censured by the court, but courts generally have no power over law licenses. We might have to wait awhile to find out if there will be any more serious repercussions. I think that in many professional settings, we might in the near future discover that some large fraction have been "faking it until they make it", but without the "making it" conclusion. The fun part is when congressional staffers use this for gigantic 10,000 page bills too large for anyone to catch it before the vote. It might already be happening.

bell-cot 3y ago

In theory, I mostly agree with you. In practice - my advice is for the academic, who is trying to get an article published in "one of the well-reputable journals". That is a weak hand to be playing. Vs. the journal's editor is in a far stronger position, to hit back hard at whoever seems to be farming out their review job to a cut-rate bot. Edit: 's/is farming/seems to be farming/'

creer 3y ago

Yes, and the point then is that it is unchecked. Would it be different if it had been farmed out to a student - and then unchecked? Or for that matter if it had been dictated - and then unchecked?

creatonez 3y ago
That could rise to the level of serious infraction too, of course. But there is one key difference that makes it worse, which is that a student is not probabilistic like an LLM is. Given enough output, no matter how well behaved you think your prompt is, the LLM will output something completely off the rails. In fact, it will give every possible output. Can happen to students too (sleep deprivation will do it), but not with the same inevitability as the LLM tool. To phrase another way, you could set up the 'human factor' in a way that you can trust unchecked output from another human (e.g. checking their expertise in academia, or if they are a commercial aircraft pilot, checking their pre-flight notes on how much sleep they got), but not for LLMs.

skywhopper 3y ago

Agree that this is a potentially system-damaging problem that will only get worse if not directly dealt with. In this case, I think the advice in the OP is good however: address the feedback from the good review, resubmit, and once the paper is accepted, then contact the editor with concerns once it's clear that you aren't objecting to needing to revise.

SubiculumCode 3y ago

but again, there is no evidence of AI, just incompetence and/or laziness.

PeterisP 3y ago

It's about informing the only person who can get extra information, decide if it's severe and do something about it - the editor. The author can't (and shouldn't) do anything directly about the anonymous reviewer, all the responsibility, authority and duty is up to the editor, who at least knows who that person is.

creatonez 3y ago
Of course. I would do pretty much the same thing (finishing the review process first before complaining), but I would mention the suspicion of AI usage. If that aspect is missed, then it's very much possible the investigation won't lead to the necessary improvement in the organization. It's worth speaking out about such a problem if you are comfortable enough in your career to risk it, because it represents something that's going to rot the entire journal and the scientific community as a whole.

NoMoreNicksLeft 3y ago

Just out of curiosity, are you aware that some lawyers have started submitting LLM-generated content as legal drafts in (US) courts of law? Last I heard, the one had been censured by the court, but courts generally have no power over law licenses. We might have to wait awhile to find out if there will be any more serious repercussions. I think that in many professional settings, we might in the near future discover that some large fraction have been "faking it until they make it", but without the "making it" conclusion. The fun part is when congressional staffers use this for gigantic 10,000 page bills too large for anyone to catch it before the vote. It might already be happening.