4 ms·
Having been a regular reviewer at ML conferences, I can tell you that it is impossible to know if the empirical results are authentic without standardized testi
by machinelearning 5y ago
Having been a regular reviewer at ML conferences, I can tell you that it is impossible to know if the empirical results are authentic without standardized testing infrastructure that controls for randomness and the datasets used.
When you're making a judgement about whether a paper is good, it is important that the paper is also true.
An inauthentic paper can present as good due to the positive empirical results it claims and an authentic theoretic paper can present as bad because of lack of empirical results or obvious utility.
Also, the grant, tenure, promotion and media are extremely relevant to being able to gather the resources to write a paper these days. You can't expect a poorly funded lab to write a polished paper unless they have ample resources to run experiments. And no matter how unbiased you think reviewers are, they are volunteers and don't feel a personal responsibility to expend a lot of effort in reviewing a paper. This is worsened if they see the other reviewers don't care either. These people are the ones who use proxies like which lab the paper came from and how polished the paper looks to make a judgement. That is how some not so good papers from a reputable lab continue to get signal boosted while other actually good contributions get overlooked. In such an environment, you need to pause to consider whether people have any reason to still do good science.