5 ms·

But how do we “solve” those incentives? There will always be winners and losers, and since successful fraud is by definition indistinguishable from genuine succ

But how do we “solve” those incentives? There will always be winners and losers, and since successful fraud is by definition indistinguishable from genuine success, there will always be people trying to game the system. The only effective solution I see is making fraud harder to hide. (There’s also making the penalty for fraud higher and trying to instill integrity, but the problem is some people will still take the risk.)

User23 3y ago
> since successful fraud is by definition indistinguishable from genuine success This is absolutely false in real science and that you say it seriously is proof of just how far gone the academy is. Needless to say no matter how much you fake your data or results, nature is not so easily fooled.
- nitwit005 3y ago
  You're misinterpreting the quote You're responding to.
scherlock 3y ago
The major issue with research is how only successful results are viewed as worth while, and study results without a "positive" out come is viewed as a failure. So this means researchers are incentized to p-hack or falsifying results since they need more funding. The reality is these negative results are just as useful and can either help other researchers avoid dead ends or they might have a not great result but with some refinement to methodology might be worthwhile for further study. Researchers should be judged on the quality of the research they perform, not their outcomes. A well crafted study that shows that further research shouldn't be done should be just a well received as a well crafted study showing more research should be done. The focus should be on the design of the study first, since no matter what the results it adds to the corpus of knowledge in a meaningful way
- armchairhacker 3y ago
  You are 100% correct. The problem is, even without the incentive of money and employment there's still the incentive of fame, and negative results simply won't get the same attention and impact as positive results. We can give people with negative results PhDs and publications, we can continue funding them. But at the end of the day, people remember and researchers cite the team who invented the LLM, not the many teams who tried and invented the less-successful models.
nwhitehead 3y ago
One idea is to put more pressure on writers of letters of recommendation. There is obviously already some rather weak "reputational incentive" to not give letters of recommendation to fraudsters. But it could be made a more formal policy. Letters could be required to include a statement about the candidate adhering to various formalized "good behavior" policies. Then when fraud happens there could be actual consequences for the letter writer. This would be like the culture in software development where you don't blame people for writing bad code, you blame reviewers for letting in bad code. (Idea comes from Angela Collier, in relation to other forms of antisocial behavior).
- jltsiren 3y ago
  1. There is already a lot of administrative bloat in the academia, and ideas like this are responsible for most of it. Every time you replace an informal process with a formal one, you add to the bloat. And the formal process rarely works as intended. 2. Letter-writers have no control over the work of the person they recommend. They cannot prevent the submission of a manuscript or a grant. They would be taking a lot of risk for no gain. 3. Letters of recommendation are a form of corruption and nepotism. If you would have a conflict of interest for reviewing a manuscript or a grant, you should not be allowed to write a letter of recommendation either.
- PeterisP 3y ago
  If what you propose would be implemented, why would anyone ever write a letter of recommendation? I'm currently doing an unpaid favor to the institution which asks for letters of recommendation, as I spend my time so they get my opinion about the candidate. If instead of thanking me for it they would want to put more pressure on me and assign actual consequences, well, that's insulting and I'd pass, I owe them literally nothing.
jdietrich 3y ago
Four words: Academic fraud is fraud. If you accept an NIH grant to conduct a rigorous study and falsify your data, then you have defrauded the federal government. If you are a supervisor, a department head or an administrator and you turn a blind eye to implausible results and indefensible practices, then you are complicit in fraud. If your journal claims to be rigorously peer-reviewed but your reviewers are rubber-stamping papers that they haven't read, then the institutions that buy very expensive subscriptions are being defrauded. Only in academia is fraud treated as a faux-pas rather than a crime. We have accepted a status quo in which academics and academic institutions play by a different set of rules to the rest of society. As a result, hundreds of millions of dollars are being wasted and patients are suffering real harm. Academic independence is valuable and should be protected, but independence does not mean impunity.

User23 3y ago

> since successful fraud is by definition indistinguishable from genuine success This is absolutely false in real science and that you say it seriously is proof of just how far gone the academy is. Needless to say no matter how much you fake your data or results, nature is not so easily fooled.

nitwit005 3y ago
You're misinterpreting the quote You're responding to.

scherlock 3y ago

The major issue with research is how only successful results are viewed as worth while, and study results without a "positive" out come is viewed as a failure. So this means researchers are incentized to p-hack or falsifying results since they need more funding. The reality is these negative results are just as useful and can either help other researchers avoid dead ends or they might have a not great result but with some refinement to methodology might be worthwhile for further study. Researchers should be judged on the quality of the research they perform, not their outcomes. A well crafted study that shows that further research shouldn't be done should be just a well received as a well crafted study showing more research should be done. The focus should be on the design of the study first, since no matter what the results it adds to the corpus of knowledge in a meaningful way

armchairhacker 3y ago
You are 100% correct. The problem is, even without the incentive of money and employment there's still the incentive of fame, and negative results simply won't get the same attention and impact as positive results. We can give people with negative results PhDs and publications, we can continue funding them. But at the end of the day, people remember and researchers cite the team who invented the LLM, not the many teams who tried and invented the less-successful models.

nwhitehead 3y ago

One idea is to put more pressure on writers of letters of recommendation. There is obviously already some rather weak "reputational incentive" to not give letters of recommendation to fraudsters. But it could be made a more formal policy. Letters could be required to include a statement about the candidate adhering to various formalized "good behavior" policies. Then when fraud happens there could be actual consequences for the letter writer. This would be like the culture in software development where you don't blame people for writing bad code, you blame reviewers for letting in bad code. (Idea comes from Angela Collier, in relation to other forms of antisocial behavior).

jltsiren 3y ago
1. There is already a lot of administrative bloat in the academia, and ideas like this are responsible for most of it. Every time you replace an informal process with a formal one, you add to the bloat. And the formal process rarely works as intended. 2. Letter-writers have no control over the work of the person they recommend. They cannot prevent the submission of a manuscript or a grant. They would be taking a lot of risk for no gain. 3. Letters of recommendation are a form of corruption and nepotism. If you would have a conflict of interest for reviewing a manuscript or a grant, you should not be allowed to write a letter of recommendation either.
PeterisP 3y ago
If what you propose would be implemented, why would anyone ever write a letter of recommendation? I'm currently doing an unpaid favor to the institution which asks for letters of recommendation, as I spend my time so they get my opinion about the candidate. If instead of thanking me for it they would want to put more pressure on me and assign actual consequences, well, that's insulting and I'd pass, I owe them literally nothing.

jdietrich 3y ago

Four words: Academic fraud is fraud. If you accept an NIH grant to conduct a rigorous study and falsify your data, then you have defrauded the federal government. If you are a supervisor, a department head or an administrator and you turn a blind eye to implausible results and indefensible practices, then you are complicit in fraud. If your journal claims to be rigorously peer-reviewed but your reviewers are rubber-stamping papers that they haven't read, then the institutions that buy very expensive subscriptions are being defrauded. Only in academia is fraud treated as a faux-pas rather than a crime. We have accepted a status quo in which academics and academic institutions play by a different set of rules to the rest of society. As a result, hundreds of millions of dollars are being wasted and patients are suffering real harm. Academic independence is valuable and should be protected, but independence does not mean impunity.