4 ms·

“…because conferences have made it mandatory to review 4-5 papers if you submit to them.” Is that really a thing? So if I submit a legit paper it is being revi

“…because conferences have made it mandatory to review 4-5 papers if you submit to them.” Is that really a thing? So if I submit a legit paper it is being reviewed forcefully by a bunch of random people from god knows where?

emil-lp 2mo ago
That's correct. And 4–5 is low-balling. I was forced to review 7 papers. Now, how are these papers chosen for you? You get to bid on which to review. Bid on 30 papers out of 30,000. I got none of those, and I had to review 7 papers I wasn't really competent enough to review.
- tgv 2mo ago
  30k papers? What field is that? How do you even select your area of competence from such a haystack? I went to smaller conferences, I suppose, but back then I also had to review papers outside my direct expertise, although nothing too remote. But I didn't know the literature well, obviously. I was able to weed out the sub-par papers (I think), but it was harder to estimate the merits of those that did make sense, as I only saw 1 or 2 per area. And that's what determines the acceptance, after all. No criticism because the reviewer judged it perfect or because the reviewer didn't know what to look for? Outcome is the same.
  - emil-lp 2mo ago
    AAAI 2026 had a record breaking 27k submissions. It doesn't scale.
- YeGoblynQueenne 2mo ago
  For me at that point it's OK to give the paper a quick read, write a few words on what it's about and whether the paper looks well written and so on, explain it's outside my area of expertise and give a weak accept with low confidence. So basically leave it up to the Area Chair. It still helps them a bit that you give the paper a quick once over and flag it up if it's somehow complete crap. The problem is that I'm probably missing the chance to review papers I'd be really interested in and that I could help to improve. I guess it's a bit ridiculous that AI conferences in particular can't find a systematic way to route papers to subject matter experts. I guess the usual system with area and sub-area keywords is overwhelmed by the extreme increase in numbers. This year AAAI made link to my Google Scholar and DBLP, link my top five papers, and add some more info about my subject to my profile; and they still failed to rate the papers that were really relevant as most relevant. At least the bidding pool was a bit smaller this time so I guess they're trying to work something out. Meanwhile submission numbers are exploding and I fear all the conferences can do is try to play catch up.
  - low_tech_love 2mo ago
    Here is my suggestion: scale the “difficulty” to the same rate as the submissions. Instruct the reviewers like this: “this year we had 10x the average submissions of the last 5 years, so be 10x stricter than you would normally.” Another idea: make it less about the written word and more about artifacts. Require more proof, more protocols, more spreadsheets, more code, etc. And stop caring about the words so much, as they really don’t mean anything anymore. Larger papers should be punished instead of rewarded; the shorter the paper (in terms of words) the higher the chance to be published. Every word should be backed up by data, code, etc. Any paragraph that doesn’t pair with proof should mean rejection.
probably_wrong 2mo ago
Speaking for the *ACL conferences, they are not "random people" in the sense that they must have "(a) at least two papers in main ACL events or Findings, plus (b) at least one more paper in the ACL Anthology or a major ML/AI venue" [1]. So they are published authors. As for it "being a thing": sadly yes. The number of papers has been steadily increasing over the years and the number of volunteer authors is simply not enough. [1] https://aclrollingreview.org/incentives2025 https://aclrollingreview.org/incentives2025
- low_tech_love 2mo ago
  Ok but what you described is very different than being forced to review if you submit. If anyone who submits a paper has to review that is just ridiculous. There is no requirement for submitting a paper.

emil-lp 2mo ago

That's correct. And 4–5 is low-balling. I was forced to review 7 papers. Now, how are these papers chosen for you? You get to bid on which to review. Bid on 30 papers out of 30,000. I got none of those, and I had to review 7 papers I wasn't really competent enough to review.

tgv 2mo ago
30k papers? What field is that? How do you even select your area of competence from such a haystack? I went to smaller conferences, I suppose, but back then I also had to review papers outside my direct expertise, although nothing too remote. But I didn't know the literature well, obviously. I was able to weed out the sub-par papers (I think), but it was harder to estimate the merits of those that did make sense, as I only saw 1 or 2 per area. And that's what determines the acceptance, after all. No criticism because the reviewer judged it perfect or because the reviewer didn't know what to look for? Outcome is the same.
- emil-lp 2mo ago
  AAAI 2026 had a record breaking 27k submissions. It doesn't scale.
YeGoblynQueenne 2mo ago
For me at that point it's OK to give the paper a quick read, write a few words on what it's about and whether the paper looks well written and so on, explain it's outside my area of expertise and give a weak accept with low confidence. So basically leave it up to the Area Chair. It still helps them a bit that you give the paper a quick once over and flag it up if it's somehow complete crap. The problem is that I'm probably missing the chance to review papers I'd be really interested in and that I could help to improve. I guess it's a bit ridiculous that AI conferences in particular can't find a systematic way to route papers to subject matter experts. I guess the usual system with area and sub-area keywords is overwhelmed by the extreme increase in numbers. This year AAAI made link to my Google Scholar and DBLP, link my top five papers, and add some more info about my subject to my profile; and they still failed to rate the papers that were really relevant as most relevant. At least the bidding pool was a bit smaller this time so I guess they're trying to work something out. Meanwhile submission numbers are exploding and I fear all the conferences can do is try to play catch up.
- low_tech_love 2mo ago
  Here is my suggestion: scale the “difficulty” to the same rate as the submissions. Instruct the reviewers like this: “this year we had 10x the average submissions of the last 5 years, so be 10x stricter than you would normally.” Another idea: make it less about the written word and more about artifacts. Require more proof, more protocols, more spreadsheets, more code, etc. And stop caring about the words so much, as they really don’t mean anything anymore. Larger papers should be punished instead of rewarded; the shorter the paper (in terms of words) the higher the chance to be published. Every word should be backed up by data, code, etc. Any paragraph that doesn’t pair with proof should mean rejection.

probably_wrong 2mo ago

Speaking for the *ACL conferences, they are not "random people" in the sense that they must have "(a) at least two papers in main ACL events or Findings, plus (b) at least one more paper in the ACL Anthology or a major ML/AI venue" [1]. So they are published authors. As for it "being a thing": sadly yes. The number of papers has been steadily increasing over the years and the number of volunteer authors is simply not enough. [1] https://aclrollingreview.org/incentives2025 https://aclrollingreview.org/incentives2025

low_tech_love 2mo ago
Ok but what you described is very different than being forced to review if you submit. If anyone who submits a paper has to review that is just ridiculous. There is no requirement for submitting a paper.