3 ms·

Given how difficult it is to know what’s true in some of these fields and what’s fiction, that might be a pretty good rate. I think the bigger question is why i

Given how difficult it is to know what’s true in some of these fields and what’s fiction, that might be a pretty good rate. I think the bigger question is why is fraud and sloppy work so important? Maybe the incentive structure of academia is the actual problem and the peer review system is an imperfect, but perhaps effective, control? What’s the alternative to peer review?

ekianjo 3y ago
> Maybe the incentive structure of academia is the actual problem it's fairly clear that it is the case. You are rewarded with tenure for pushing lots of papers, regardless of their validity, since nobody actually cares about replicating anything. > peer review system is an imperfect, but perhaps effective The peer review system is just about checking points of details or calculation in a study, they never bother to actually check if the original results are valid. It's a mostly useless process that does nothing but remove the most obvious problems in the writing stage. > the alternative just like we did for hundreds of years now, proper replication baked in, by different people in different places, following the exact same conditions or protocol.
- fnordpiglet 3y ago
  Some (many) experiments the replication is difficult it exceptionally expensive. Particularly for experiments with large exotic equipment, huge populations, etc. But certainly replication should be incentivized - sadly I think even then it’ll be largely a secondary behavior.
mike_hearn 3y ago
> Maybe the incentive structure of academia is the actual problem There are definitely major problems in academic research (the number of retracted papers should be much higher than it is), but it's worth noting that Tessier-Lavigne did the non-reproducible research whilst working at Genentech. One of the strangest parts of this story is how Genentech apparently already knew that the claims didn't replicate, but allowed MTL to publish a paper in Nature about it anyway ... and then went ahead and presented the claims to investors. MTL left soon after Genentech was acquired, and when they concluded years later that the paper was definitely bad they had apparently lost the ability to retract it, even though it had been done by their own employees. Whilst papers are not much of anything, presenting research to a potential acquirer that you knew wasn't reliable sounds like the basis of potential legal claims. Normally companies are very careful about that because the liability from misleading investors or acquirers can be huge.

ekianjo 3y ago

> Maybe the incentive structure of academia is the actual problem it's fairly clear that it is the case. You are rewarded with tenure for pushing lots of papers, regardless of their validity, since nobody actually cares about replicating anything. > peer review system is an imperfect, but perhaps effective The peer review system is just about checking points of details or calculation in a study, they never bother to actually check if the original results are valid. It's a mostly useless process that does nothing but remove the most obvious problems in the writing stage. > the alternative just like we did for hundreds of years now, proper replication baked in, by different people in different places, following the exact same conditions or protocol.

fnordpiglet 3y ago
Some (many) experiments the replication is difficult it exceptionally expensive. Particularly for experiments with large exotic equipment, huge populations, etc. But certainly replication should be incentivized - sadly I think even then it’ll be largely a secondary behavior.

mike_hearn 3y ago

> Maybe the incentive structure of academia is the actual problem There are definitely major problems in academic research (the number of retracted papers should be much higher than it is), but it's worth noting that Tessier-Lavigne did the non-reproducible research whilst working at Genentech. One of the strangest parts of this story is how Genentech apparently already knew that the claims didn't replicate, but allowed MTL to publish a paper in Nature about it anyway ... and then went ahead and presented the claims to investors. MTL left soon after Genentech was acquired, and when they concluded years later that the paper was definitely bad they had apparently lost the ability to retract it, even though it had been done by their own employees. Whilst papers are not much of anything, presenting research to a potential acquirer that you knew wasn't reliable sounds like the basis of potential legal claims. Normally companies are very careful about that because the liability from misleading investors or acquirers can be huge.