3 ms·

I think the scale of the problem varies wildly between different fields of science. For instance, in computer science, double blind reviewing is the norm, and

by hedora 2y ago

I think the scale of the problem varies wildly between different fields of science. For instance, in computer science, double blind reviewing is the norm, and the biggest issue is that there is an incentive to produce more papers with smaller contributions, instead of a mixture of larger, better, papers, and 1-2 column notes that describe important, but simple results (such notes used to be published routinely). In other fields, not only are submissions not anonymous, but editors sometimes override the reviewers (in both directions), for what appear to be political purposes.

Der_Einzige 2y ago
Uhh, the elephant in the room for CS/ML publications is that a lot of people hardcore lie on their results page since almost no one tries to reproduce anything. I've witnessed folks straight out of their undergrad with literally 1 workshop paper at an ACL track conference get thrown 150K starting offers by no-name startups. Imagine how important it must be to get your first NeurIPS main conference publication. The ROI likely amounts to 300K+ per year, so if anything, I'm surprised that cheating/lies in the academy aren't even more brazen. Dynamics like this are why both Stanford and Harvard's presidents had to step-down for gross academic plagerism/misconduct. 1. https://www.science.org/doi/10.1126/science.359.6377.725 https://www.science.org/doi/10.1126/science.359.6377.725 (2018) 2. https://www.technologyreview.com/2020/11/12/1011944/artificial-intelligence-replication-crisis-science-big-tech-google-deepmind-facebook-openai/ https://www.technologyreview.com/2020/11/12/1011944/artifici... (2020) 3. https://www.nature.com/articles/d41586-023-03817-6 https://www.nature.com/articles/d41586-023-03817-6 (2023, Nature)
- godelski 2y ago
  I've seen it happen even when source is available. I've also seen a lot of really terrible math mistakes in ML because people don't understand the underlying assumptions of the metrics they use. The rot is everywhere because people forgot the goal. We all get caught up in the metrics that are used to judge us that we forgot what we were proxying in the first place.
jltsiren 2y ago
Double blind reviewing is not the norm in computer science. You will sometimes see it, but it depends on the subfield, the specific conference/journal, and even the year. In my experience as an author/reviewer, the submissions have been anonymous maybe 10% of the time. And the biggest issue I see in CS is the focus on gatekeeping. When many (most?) papers are published in conferences with a single round of reviews, peer reviewing becomes more about accepting/rejecting the paper than improving it. The average review I receive in bioinformatics is more useful to me than the average review in CS.

by hedora 2y ago

Der_Einzige 2y ago

Uhh, the elephant in the room for CS/ML publications is that a lot of people hardcore lie on their results page since almost no one tries to reproduce anything. I've witnessed folks straight out of their undergrad with literally 1 workshop paper at an ACL track conference get thrown 150K starting offers by no-name startups. Imagine how important it must be to get your first NeurIPS main conference publication. The ROI likely amounts to 300K+ per year, so if anything, I'm surprised that cheating/lies in the academy aren't even more brazen. Dynamics like this are why both Stanford and Harvard's presidents had to step-down for gross academic plagerism/misconduct. 1. https://www.science.org/doi/10.1126/science.359.6377.725 https://www.science.org/doi/10.1126/science.359.6377.725 (2018) 2. https://www.technologyreview.com/2020/11/12/1011944/artificial-intelligence-replication-crisis-science-big-tech-google-deepmind-facebook-openai/ https://www.technologyreview.com/2020/11/12/1011944/artifici... (2020) 3. https://www.nature.com/articles/d41586-023-03817-6 https://www.nature.com/articles/d41586-023-03817-6 (2023, Nature)

godelski 2y ago
I've seen it happen even when source is available. I've also seen a lot of really terrible math mistakes in ML because people don't understand the underlying assumptions of the metrics they use. The rot is everywhere because people forgot the goal. We all get caught up in the metrics that are used to judge us that we forgot what we were proxying in the first place.

jltsiren 2y ago

Double blind reviewing is not the norm in computer science. You will sometimes see it, but it depends on the subfield, the specific conference/journal, and even the year. In my experience as an author/reviewer, the submissions have been anonymous maybe 10% of the time. And the biggest issue I see in CS is the focus on gatekeeping. When many (most?) papers are published in conferences with a single round of reviews, peer reviewing becomes more about accepting/rejecting the paper than improving it. The average review I receive in bioinformatics is more useful to me than the average review in CS.