3 ms·
Reviewing is also becoming an utter mess in hyped areas such as AI. It's unpaid, thankless work, where most of what you must evaluate is increasingly some gamed
by bonoboTP 1y ago
Reviewing is also becoming an utter mess in hyped areas such as AI. It's unpaid, thankless work, where most of what you must evaluate is increasingly some gamed, partially AI-generated paper with strategically hidden aspects, rampant cherry picking, etc. Conference sizes are exploding. AAAI went from 15k submissions last year to 30k submissions, with 75k unique authors. There are close to 30k reviewers. It's becoming an assembly line factory with people not knowing each other any more (even in an anonymous review process, if you value the community, you behave differently). It's similar to what old professors lament when referring back to university 50 years ago. When groups were so small, profs knew individually the students. Then came the sausage factory model with 10x the students, students becoming just an ID number, reduction of oral exams in favor or written, gaming the rules more, increased cheating etc.
The academic conference situation will implode under its own weight. As more and more people enter, the quality of reviews suffers, there is less and less illusions about any form of duty, responsibility, honor etc. and it's all a game of turf wars, citations rings, just slipping through the cracks by submitting enormous numbers of papers and resubmitting in a few months those that get rejected, without even fixing the typos that the first reviewers pointed out. And observing this just makes the few who did care also become jaded and put in less energy. Just as you have people in college who want to put in the absolute minimum work in order to get the piece of paper, you have the same happening in academia. Pump out the most papers with the least works, and the reviewers are just an obstacle in that view.
- godelski 1y agoTell me about it... I almost didn't graduate because of this stupidity... I can say that the review process wasn't great when I started but there was a quick decline after GPT 3 came out. Everything became so short sighted and I've (and many others) even been screaming about this wall we were going to hit for years. We absolutely could have avoided this slowdown if we just understood you can put your eggs in more than one basket and stopped saying "fuck theorists". In betting on "scale is all you need" they destroyed the upstream research pipeline that got us to where we are today... And then we can't figure out that there'll be an exponential increase in papers if we keep adding more people, publish 5 papers a year, and reject 80% of them. How do you expect this to be sustainable? I think at a minimum academia needs to start tracking abuse. Because right now, when we find plagiarism we hide it, usually just disappearing during blind review. These people should get a one year ban on first offense and a 5 year ban for every subsequent occurrence. Make it a citation network, and everyone you've co-authored with in the last year gets flagged for extra review. And FFS, let's not give best fucking paper of the year at the number one ML conference to a fucking intern who hacked the company you interned at. Not only is that just encouraging more highly unethical behavior, but how the fuck can you even trust the results of that paper? I love this field, but how did we get infiltrated by a cult?