Peer review is flawed, but I don't think it's broken. For context, I'm a late-year PhD student in machine learning theory, and I submit almost entirely to confe

Peer review is flawed, but I don't think it's broken. For context, I'm a late-year PhD student in machine learning theory, and I submit almost entirely to conferences, which is the norm in my field. To me, the big problem is that there is little incentive for experts to review papers. I write a few papers every year, and I review 5x that number. Slightly more senior people -- postdocs and junior faculty -- may review more like 30-40 papers. Most papers I review are not good papers. This is especially true at the big machine learning conferences like NeurIPS, ICML, and ICLR, where a disconcertingly large fraction (1/3?) of submissions are outright bad. Most of them are at best "meh". So there's not much benefit to the reviewer beyond some notion of community service. I think my ~colleagues (PhD students, postdocs, young faculty) have similar opinions. We've certainly discussed these issues in the past. One positive step NeurIPS has taken is giving "best reviewer" awards that come from with a free registration. That's a nice benefit, even if it won't on its own incentivize good reviews. Some people have suggested outright paying people for reviews at some quasi-competitive rate, maybe $100 per review (note that this is not great -- a good review takes at least several hours if the paper is not terrible). NeurIPS also added a requirement this year that all submitting authors must review, which is an interesting experiment. My impractical wish is that we also have some kind of "worst reviewer" sticker for all the reviewers who just paraphrase the abstract, add some milquetoast feelings on novelty, and weak reject. Less facetiously, some kind of StackOverflow reputation system might be useful. As is there's so little external signal about being a good or bad reviewer that the incentives are pretty wrong. Some kind of points system might help. But my overall point might be: reviewers by and large aren't malicious, jealous people who are only rejecting your work because they want to protect their own turf. They are more likely good faith actors who are trying to balance time spent on a pretty thankless but societally useful job (reviewing) with time spent on much more personally enjoyable and rewarding activities (research, family, hobbies, etc). There aren't really obvious solutions to tilt the balance toward better reviews that can still handle the volume of papers.