3 ms·

> On the whole, I don't see how if the authors were academics with Tenure, they'd survive this one. This is entirely NOT how it's meant to work. WDYM?

ggm 3y ago
If you submit a peer review paper to a frontline journal like NATURE and then it turns out your work cannot be reproduced, and you do not either issue a retraction (and explanation) or an explation how to make it work, and your data, then you stand very likely to be accused of falsifying data. If you had secured tenure at a uni because of achieving the nature paper, and this happened, your head of department would quietly suggest you look elsewhere for employment because you won't be getting seniority in the department, ever. You can publish shit in the minor outer planet regional journal of skeptical telekenesis, and nobody cares. If you publish a paper in Nature "we cured cancer" and it turns out you forgot to stir the lab reagents before adding them to the PCR machine, people care.
- sashank_1509 3y ago
  not true, no idea where you got the idea that academics have such high standards, you might be referring to some long lost past that no longer exists. I have spent time in ML academia and standards are mostly a joke but things self select in the sense that often papers that are cited well and generate a buzz are those that are easily reproducible and show a significant improvement, (since researchers instantly try it and it works). But tuning baselines to look bad, creating your own benchmarks, modifying experiment conditions to make your method look good is par for the course for ML, this paper is simply far more egregious than average. I think it’s honestly worse in other fields like cancer research where apparently >50% of the top cited papers don’t replicate. Read about the “Replication crisis” EDIT: some more thoughts on issues. There really are 2 kinds of citations in research papers. One is the good kind of citation, that occurs when your method is so good that others use it and cite it. This unfortunately is very hard to do, even if your method is good it needs to be packaged well and researchers aren’t great SWEs. The second kind of citation is when, you get cited in related work of another paper that just states so and so method achieved so and so results on so and so benchmark. This kind of citation creates bad incentives, your goal is to be on top of some benchmark no matter what. So you start inventing benchmarks, tuning baselines to look bad etc. this citation is also easier to get so a majority of papers try to go for this instead. Maybe a solution is to just ban related works sections but who knows if that will work.
- maximilianroos 3y ago
  > If you had secured tenure at a uni because of achieving the nature paper, and this happened, your head of department would quietly suggest you look elsewhere for employment But if you have tenure then you can't be fired, no? Isn't that the role of tenure? Presumably you're in the bad situation if you _don't_ have tenure?
- AndrewKemendo 3y ago
  You wish Academics are as unethical as businessmen - and especially the overlap between tech and science research. As long as these researchers are bringing in grant money and getting press coverage (good or bad) they will keep their jobs or will be promoted.

ggm 3y ago

If you submit a peer review paper to a frontline journal like NATURE and then it turns out your work cannot be reproduced, and you do not either issue a retraction (and explanation) or an explation how to make it work, and your data, then you stand very likely to be accused of falsifying data. If you had secured tenure at a uni because of achieving the nature paper, and this happened, your head of department would quietly suggest you look elsewhere for employment because you won't be getting seniority in the department, ever. You can publish shit in the minor outer planet regional journal of skeptical telekenesis, and nobody cares. If you publish a paper in Nature "we cured cancer" and it turns out you forgot to stir the lab reagents before adding them to the PCR machine, people care.

sashank_1509 3y ago
not true, no idea where you got the idea that academics have such high standards, you might be referring to some long lost past that no longer exists. I have spent time in ML academia and standards are mostly a joke but things self select in the sense that often papers that are cited well and generate a buzz are those that are easily reproducible and show a significant improvement, (since researchers instantly try it and it works). But tuning baselines to look bad, creating your own benchmarks, modifying experiment conditions to make your method look good is par for the course for ML, this paper is simply far more egregious than average. I think it’s honestly worse in other fields like cancer research where apparently >50% of the top cited papers don’t replicate. Read about the “Replication crisis” EDIT: some more thoughts on issues. There really are 2 kinds of citations in research papers. One is the good kind of citation, that occurs when your method is so good that others use it and cite it. This unfortunately is very hard to do, even if your method is good it needs to be packaged well and researchers aren’t great SWEs. The second kind of citation is when, you get cited in related work of another paper that just states so and so method achieved so and so results on so and so benchmark. This kind of citation creates bad incentives, your goal is to be on top of some benchmark no matter what. So you start inventing benchmarks, tuning baselines to look bad etc. this citation is also easier to get so a majority of papers try to go for this instead. Maybe a solution is to just ban related works sections but who knows if that will work.
maximilianroos 3y ago
> If you had secured tenure at a uni because of achieving the nature paper, and this happened, your head of department would quietly suggest you look elsewhere for employment But if you have tenure then you can't be fired, no? Isn't that the role of tenure? Presumably you're in the bad situation if you _don't_ have tenure?
AndrewKemendo 3y ago
You wish Academics are as unethical as businessmen - and especially the overlap between tech and science research. As long as these researchers are bringing in grant money and getting press coverage (good or bad) they will keep their jobs or will be promoted.