2 ms·

Just a comment: it's nice to step back and think about what papers are for. I think a useful aim is to describe new findings that can help advance the field. Th

Just a comment: it's nice to step back and think about what papers are for. I think a useful aim is to describe new findings that can help advance the field. There may be occasional "this is cool but probably useless" results, but one would expect a good paper to be one that another researcher looks at and gets inspired or boosted in their own work. My experience as a researcher is that these good papers are very easy to spot in your own field. So really I think that's what's more important, is finding the good ones, and then presumably citing them as a measure of their utility. The question of cheating of cherry picking or any kind of subterfuge is only really relevant to grant and tenure & promotion committees, media, and other external assessments. For the supposed target audience of papers, it is clear as day whether they are describing something useful or just trying to publish something

machinelearning 5y ago
Having been a regular reviewer at ML conferences, I can tell you that it is impossible to know if the empirical results are authentic without standardized testing infrastructure that controls for randomness and the datasets used. When you're making a judgement about whether a paper is good, it is important that the paper is also true. An inauthentic paper can present as good due to the positive empirical results it claims and an authentic theoretic paper can present as bad because of lack of empirical results or obvious utility. Also, the grant, tenure, promotion and media are extremely relevant to being able to gather the resources to write a paper these days. You can't expect a poorly funded lab to write a polished paper unless they have ample resources to run experiments. And no matter how unbiased you think reviewers are, they are volunteers and don't feel a personal responsibility to expend a lot of effort in reviewing a paper. This is worsened if they see the other reviewers don't care either. These people are the ones who use proxies like which lab the paper came from and how polished the paper looks to make a judgement. That is how some not so good papers from a reputable lab continue to get signal boosted while other actually good contributions get overlooked. In such an environment, you need to pause to consider whether people have any reason to still do good science.

machinelearning 5y ago

Having been a regular reviewer at ML conferences, I can tell you that it is impossible to know if the empirical results are authentic without standardized testing infrastructure that controls for randomness and the datasets used. When you're making a judgement about whether a paper is good, it is important that the paper is also true. An inauthentic paper can present as good due to the positive empirical results it claims and an authentic theoretic paper can present as bad because of lack of empirical results or obvious utility. Also, the grant, tenure, promotion and media are extremely relevant to being able to gather the resources to write a paper these days. You can't expect a poorly funded lab to write a polished paper unless they have ample resources to run experiments. And no matter how unbiased you think reviewers are, they are volunteers and don't feel a personal responsibility to expend a lot of effort in reviewing a paper. This is worsened if they see the other reviewers don't care either. These people are the ones who use proxies like which lab the paper came from and how polished the paper looks to make a judgement. That is how some not so good papers from a reputable lab continue to get signal boosted while other actually good contributions get overlooked. In such an environment, you need to pause to consider whether people have any reason to still do good science.