not true, no idea where you got the idea that academics have such high standards, you might be referring to some long lost past that no longer exists. I have sp

not true, no idea where you got the idea that academics have such high standards, you might be referring to some long lost past that no longer exists. I have spent time in ML academia and standards are mostly a joke but things self select in the sense that often papers that are cited well and generate a buzz are those that are easily reproducible and show a significant improvement, (since researchers instantly try it and it works). But tuning baselines to look bad, creating your own benchmarks, modifying experiment conditions to make your method look good is par for the course for ML, this paper is simply far more egregious than average. I think it’s honestly worse in other fields like cancer research where apparently >50% of the top cited papers don’t replicate. Read about the “Replication crisis” EDIT: some more thoughts on issues. There really are 2 kinds of citations in research papers. One is the good kind of citation, that occurs when your method is so good that others use it and cite it. This unfortunately is very hard to do, even if your method is good it needs to be packaged well and researchers aren’t great SWEs. The second kind of citation is when, you get cited in related work of another paper that just states so and so method achieved so and so results on so and so benchmark. This kind of citation creates bad incentives, your goal is to be on top of some benchmark no matter what. So you start inventing benchmarks, tuning baselines to look bad etc. this citation is also easier to get so a majority of papers try to go for this instead. Maybe a solution is to just ban related works sections but who knows if that will work.