3 ms·

For me at that point it's OK to give the paper a quick read, write a few words on what it's about and whether the paper looks well written and so on, explain it

For me at that point it's OK to give the paper a quick read, write a few words on what it's about and whether the paper looks well written and so on, explain it's outside my area of expertise and give a weak accept with low confidence. So basically leave it up to the Area Chair. It still helps them a bit that you give the paper a quick once over and flag it up if it's somehow complete crap. The problem is that I'm probably missing the chance to review papers I'd be really interested in and that I could help to improve. I guess it's a bit ridiculous that AI conferences in particular can't find a systematic way to route papers to subject matter experts. I guess the usual system with area and sub-area keywords is overwhelmed by the extreme increase in numbers. This year AAAI made link to my Google Scholar and DBLP, link my top five papers, and add some more info about my subject to my profile; and they still failed to rate the papers that were really relevant as most relevant. At least the bidding pool was a bit smaller this time so I guess they're trying to work something out. Meanwhile submission numbers are exploding and I fear all the conferences can do is try to play catch up.

low_tech_love 2mo ago
Here is my suggestion: scale the “difficulty” to the same rate as the submissions. Instruct the reviewers like this: “this year we had 10x the average submissions of the last 5 years, so be 10x stricter than you would normally.” Another idea: make it less about the written word and more about artifacts. Require more proof, more protocols, more spreadsheets, more code, etc. And stop caring about the words so much, as they really don’t mean anything anymore. Larger papers should be punished instead of rewarded; the shorter the paper (in terms of words) the higher the chance to be published. Every word should be backed up by data, code, etc. Any paragraph that doesn’t pair with proof should mean rejection.

low_tech_love 2mo ago

Here is my suggestion: scale the “difficulty” to the same rate as the submissions. Instruct the reviewers like this: “this year we had 10x the average submissions of the last 5 years, so be 10x stricter than you would normally.” Another idea: make it less about the written word and more about artifacts. Require more proof, more protocols, more spreadsheets, more code, etc. And stop caring about the words so much, as they really don’t mean anything anymore. Larger papers should be punished instead of rewarded; the shorter the paper (in terms of words) the higher the chance to be published. Every word should be backed up by data, code, etc. Any paragraph that doesn’t pair with proof should mean rejection.