4 ms·

I think we need to rethink the core concept of Universities having publish or perish as a part of being a Professor. The reason why fraud exists is because of t

I think we need to rethink the core concept of Universities having publish or perish as a part of being a Professor. The reason why fraud exists is because of the incentive structure and due to the nature of research you have intelligent people that will game the system. So, we have to figure out how to change the incentive structure so that professors don't attempt to perform Academic Fraud or we h ave to figure out if fraud does exist after the fact. I have tried to reproduce ML / Deep learning research. I have a few heuristics that don't really tell me if fraud exists but either they just can't release the data due to special circumstances that I can understand or it isn't kosher. The biggest one is that you have to register or ask them for their test data. Another one is that the paper has been published for more than two years and there are no citations from other work.

bonoboTP 5y ago
Keeping test data private is good practice actually. It keeps from hyperparameter optimization on the test set. Yes theoretically the dataset authors might still do it, but it's still better for the field as a whole to keep some test data secret and only use it for evaluation. Registering before getting data is common, so they can make you agree to some terms. It's not suspicious in itself. There are no hard and fast rules of thumb. You can look at the group, the people involved, the country, the writing style, the quality and thoroughness of ablation, is there code available etc. But nothing is 100%.
ModernMech 5y ago
> So, we have to figure out how to change the incentive structure so that professors don't attempt to perform Academic Fraud or we h ave to figure out if fraud does exist after the fact. One possible way: increase funding to the arts and sciences. The total NSF budget was ~$8B, or 0.03% of US GDP. We could double that and no one would even notice. Increased funding means more money to go around, means higher acceptance rates for grants, means less incentive to write fraudulent papers. The best part is, you have the power to actually change this. Write your congressperson, vote for a congressperson who pledges to do this, run for congress yourself, or organize other people to do this with you.
- zitterbewegung 5y ago
  Increasing the budget doesn’t solve the problem. You would need to create a set of well funded auditors that have domain experience but the problem with that is no one would want the job.

bonoboTP 5y ago

Keeping test data private is good practice actually. It keeps from hyperparameter optimization on the test set. Yes theoretically the dataset authors might still do it, but it's still better for the field as a whole to keep some test data secret and only use it for evaluation. Registering before getting data is common, so they can make you agree to some terms. It's not suspicious in itself. There are no hard and fast rules of thumb. You can look at the group, the people involved, the country, the writing style, the quality and thoroughness of ablation, is there code available etc. But nothing is 100%.

ModernMech 5y ago

> So, we have to figure out how to change the incentive structure so that professors don't attempt to perform Academic Fraud or we h ave to figure out if fraud does exist after the fact. One possible way: increase funding to the arts and sciences. The total NSF budget was ~$8B, or 0.03% of US GDP. We could double that and no one would even notice. Increased funding means more money to go around, means higher acceptance rates for grants, means less incentive to write fraudulent papers. The best part is, you have the power to actually change this. Write your congressperson, vote for a congressperson who pledges to do this, run for congress yourself, or organize other people to do this with you.

zitterbewegung 5y ago
Increasing the budget doesn’t solve the problem. You would need to create a set of well funded auditors that have domain experience but the problem with that is no one would want the job.