3 ms·

Elimination of venue-based reviews is an interesting proposition. I’ll have to think on that. The contrast between ML and physics is equally interesting, and s

Elimination of venue-based reviews is an interesting proposition. I’ll have to think on that. The contrast between ML and physics is equally interesting, and seems to corroborate the experience of another poster, who expresses doubt about removing peer-review from long-haul studies in biological sciences. I’m still struggling to put a finger on why exactly this difference might exist, though. Anyway, thanks for your thoughts :)

godelski 3y ago
The value of venues I see as facilitating networking and collaboration. Maybe these could be centered around invited speakers and presenters? For the difference between ML and physics/other sciences, there are a few points that I see. - First, ML submits to conferences rather than journals. This means typically you have one shot at acceptance, and possibly one chance to rebuttal. Other fields use journals, where there often is a push to accept and that the reviewer's job is to make the paper better rather than filter absolutely. This results in no one actually advocating for the paper to be published other than the authors (no incentive to accept). Worse, we have a zero sum game, so if reviewers are also authors of other works (common) they have an optimal strategy of rejecting other players' works (large incentive to reject). - Second, ML is extremely hype driven. The field just moves lightning fast, it is insane. This leads to incremental improvements (technically true for all science) but muddies the waters since it is easy to reject on low novelty but if you don't release your work fast you get scooped and get rejected for again low novelty. Unfortunately we don't define concurrent work well (and I've had works actually rejected because we did not cite works that were publicly released (arxiv) around the same time as us or, worse, after we submitted to the conference!). - Third, there are a large number of people submitting works (nearly 10k per conference), the field is wide (reducing domain expertise), and highly empirical (few learn theory and base understandings and instead rely on benchmark results to evaluate). The size brings with it a lot of noise and the empirical nature reduces rigor (noise in evaluation). The large number of works also results in low reviewer quality as chairs scramble to add reviewers to works (in my last review, 3 of 4 reviewers self reported 3/5 confidence scores, meaning they did not work in this domain). So we just have way more sources of noise than would be seen in many other fields. Consistency experiments support this as well: finding reviewers are good at rejecting bad works but bad at rejecting good works. Which concludes that reviewers over reject (default strategy). I also appreciate your opinions. I certainly don't have all the answers. As I see it, we got to decide as a community. The only way to do that is to hear complaints, proposals, and challenges. We're all in this together so to me it is most important that we openly discuss these. Thank you for your thoughts and comments.

godelski 3y ago

The value of venues I see as facilitating networking and collaboration. Maybe these could be centered around invited speakers and presenters? For the difference between ML and physics/other sciences, there are a few points that I see. - First, ML submits to conferences rather than journals. This means typically you have one shot at acceptance, and possibly one chance to rebuttal. Other fields use journals, where there often is a push to accept and that the reviewer's job is to make the paper better rather than filter absolutely. This results in no one actually advocating for the paper to be published other than the authors (no incentive to accept). Worse, we have a zero sum game, so if reviewers are also authors of other works (common) they have an optimal strategy of rejecting other players' works (large incentive to reject). - Second, ML is extremely hype driven. The field just moves lightning fast, it is insane. This leads to incremental improvements (technically true for all science) but muddies the waters since it is easy to reject on low novelty but if you don't release your work fast you get scooped and get rejected for again low novelty. Unfortunately we don't define concurrent work well (and I've had works actually rejected because we did not cite works that were publicly released (arxiv) around the same time as us or, worse, after we submitted to the conference!). - Third, there are a large number of people submitting works (nearly 10k per conference), the field is wide (reducing domain expertise), and highly empirical (few learn theory and base understandings and instead rely on benchmark results to evaluate). The size brings with it a lot of noise and the empirical nature reduces rigor (noise in evaluation). The large number of works also results in low reviewer quality as chairs scramble to add reviewers to works (in my last review, 3 of 4 reviewers self reported 3/5 confidence scores, meaning they did not work in this domain). So we just have way more sources of noise than would be seen in many other fields. Consistency experiments support this as well: finding reviewers are good at rejecting bad works but bad at rejecting good works. Which concludes that reviewers over reject (default strategy). I also appreciate your opinions. I certainly don't have all the answers. As I see it, we got to decide as a community. The only way to do that is to hear complaints, proposals, and challenges. We're all in this together so to me it is most important that we openly discuss these. Thank you for your thoughts and comments.