3 ms·
I mostly agree with your assessment of peer review, but feel that it's gone too far. It's decades of metric hacking going unresolved. My research area is ML and
by godelski 3y ago
I mostly agree with your assessment of peer review, but feel that it's gone too far. It's decades of metric hacking going unresolved. My research area is ML and it is definitely far out of hand. When I was in physics I'd probably be closer to your statement. But right now we have measurements that conclude "reviewers are good at identifying bad papers, but bad at identifying good papers." Either I think we need to accept more works (as long as they are valid and useful) and/or reducing the value of a top tier venue (which right now is top or none). I think papers should get multiple rounds, as our goal is to improve works, and submissions should always be rolling. I think this can help, because I agree that peer review should be about eliminating the worst, not identifying the best (far too difficult, and hundreds of years of evidence).
I think one important thing that venues could do is host data and code/tools for works. Harddrive space is rather cheap now (especially with tax benefits and donations) and can only result in high value to the community. The other thing is having formats like OpenReview, where works can continuously be discussed in the open. With authors and others being able to defend/criticize/question works. But I think there should be some filter, even if low, and rules for control of quality.
For the most part, I do actually think getting rid of venue based review would be a step in the right direction. I use these words because I like to encourage the idea that open publication still leads to peer reviewing. In ML a lot is done with arxiv + twitter, I just think we should better formalize this. It allows for a lot of freedom and for high speed. It has problems, but I don't see them as any worse than the current system (I see an overall decrease in problems). Good science requires risks and a lot of creativity. The modern word advantage is that we have higher numbers of researchers and monte carlo sampling in parallel helps optimize. You want to encourage tail end samples too to escape local minima. The history of science is a history of upsets. You don't get upsets by doing what everyone is doing. I agree that top down hierarchies discourage such thinking and are an overall net negative to science and advancing human knowledge. Creativity is critical.
For predictions, I agree. I lean towards predicting disruption, but I'm not even certain of that. And disruption can go many different ways. I'm glad we're starting the discussions, but I think they need to be deeper and more honest.
- omginternets 3y agoElimination of venue-based reviews is an interesting proposition. I’ll have to think on that. The contrast between ML and physics is equally interesting, and seems to corroborate the experience of another poster, who expresses doubt about removing peer-review from long-haul studies in biological sciences. I’m still struggling to put a finger on why exactly this difference might exist, though. Anyway, thanks for your thoughts :)
- godelski 3y agoThe value of venues I see as facilitating networking and collaboration. Maybe these could be centered around invited speakers and presenters? For the difference between ML and physics/other sciences, there are a few points that I see. - First, ML submits to conferences rather than journals. This means typically you have one shot at acceptance, and possibly one chance to rebuttal. Other fields use journals, where there often is a push to accept and that the reviewer's job is to make the paper better rather than filter absolutely. This results in no one actually advocating for the paper to be published other than the authors (no incentive to accept). Worse, we have a zero sum game, so if reviewers are also authors of other works (common) they have an optimal strategy of rejecting other players' works (large incentive to reject). - Second, ML is extremely hype driven. The field just moves lightning fast, it is insane. This leads to incremental improvements (technically true for all science) but muddies the waters since it is easy to reject on low novelty but if you don't release your work fast you get scooped and get rejected for again low novelty. Unfortunately we don't define concurrent work well (and I've had works actually rejected because we did not cite works that were publicly released (arxiv) around the same time as us or, worse, after we submitted to the conference!). - Third, there are a large number of people submitting works (nearly 10k per conference), the field is wide (reducing domain expertise), and highly empirical (few learn theory and base understandings and instead rely on benchmark results to evaluate). The size brings with it a lot of noise and the empirical nature reduces rigor (noise in evaluation). The large number of works also results in low reviewer quality as chairs scramble to add reviewers to works (in my last review, 3 of 4 reviewers self reported 3/5 confidence scores, meaning they did not work in this domain). So we just have way more sources of noise than would be seen in many other fields. Consistency experiments support this as well: finding reviewers are good at rejecting bad works but bad at rejecting good works. Which concludes that reviewers over reject (default strategy). I also appreciate your opinions. I certainly don't have all the answers. As I see it, we got to decide as a community. The only way to do that is to hear complaints, proposals, and challenges. We're all in this together so to me it is most important that we openly discuss these. Thank you for your thoughts and comments.