3 ms·

This already happens. It's clear when a reviewer used an LLM, and it's very annoying for the authors that have to respond to what are usually low quality, super

This already happens. It's clear when a reviewer used an LLM, and it's very annoying for the authors that have to respond to what are usually low quality, superficial reviews.

CrazyStat 7d ago
To be fair, low quality superficial reviews were also not uncommon before LLMs…
mlmonkey 7d ago
Boy do I have news for you! :-D "low quality, superficial reviews" have always been around. Reviewing is most often an unpaid, thankless job and many times reviewers barely put in the effort.
- __lm__ 6d ago
  There is still a difference. Previously, if the review was low quality and superficial, it was also quite short and easy to answer. Now I receive fully LLM-written reviews of 2-3 pages (or more) of superficial comments asking for tons of additional text and experiments that are almost always outside the scope of the work. Answering such a review requires a lot of work with absolutely no gain.
  - mlmonkey 5d ago
    I hear you. I just happened to submit a paper to Neurips this year, and the "reviewer" sent back so many questions, etc. that the character limit of the response could never accomodate my response. So I just addressed a few of the main points and called it a day. I am sure an LLM review could be made much shorter with some prompting.
    - maleldil 5d ago
      The character limit thing is real. Another problem is that even if you manage to address all of their points, they're not required to engage with that. They can just ask the LLM to write a basic response back and say they won't change the score.
- tomsmeding 6d ago
  Similar experience to sibling: a disinterested human isn't going to ask for 30 different detailed things to add to, or change in, the paper that could technically be improved but aren't worth the additional page real estate. An LLM reviewer definitely does this.

3 ms·

This already happens. It's clear when a reviewer used an LLM, and it's very annoying for the authors that have to respond to what are usually low quality, super

by maleldil 7d ago

This already happens. It's clear when a reviewer used an LLM, and it's very annoying for the authors that have to respond to what are usually low quality, superficial reviews.

CrazyStat 7d ago
To be fair, low quality superficial reviews were also not uncommon before LLMs…
mlmonkey 7d ago
Boy do I have news for you! :-D "low quality, superficial reviews" have always been around. Reviewing is most often an unpaid, thankless job and many times reviewers barely put in the effort.
- __lm__ 6d ago
  There is still a difference. Previously, if the review was low quality and superficial, it was also quite short and easy to answer. Now I receive fully LLM-written reviews of 2-3 pages (or more) of superficial comments asking for tons of additional text and experiments that are almost always outside the scope of the work. Answering such a review requires a lot of work with absolutely no gain.
  - mlmonkey 5d ago
    I hear you. I just happened to submit a paper to Neurips this year, and the "reviewer" sent back so many questions, etc. that the character limit of the response could never accomodate my response. So I just addressed a few of the main points and called it a day. I am sure an LLM review could be made much shorter with some prompting.
    - maleldil 5d ago
      The character limit thing is real. Another problem is that even if you manage to address all of their points, they're not required to engage with that. They can just ask the LLM to write a basic response back and say they won't change the score.
- tomsmeding 6d ago
  Similar experience to sibling: a disinterested human isn't going to ask for 30 different detailed things to add to, or change in, the paper that could technically be improved but aren't worth the additional page real estate. An LLM reviewer definitely does this.