5 ms·

> This retraction is Tessier-Lavigne’s fourth in as many months, a stunning turn of events for a researcher of his stature. That says a lot about how worthless

> This retraction is Tessier-Lavigne’s fourth in as many months, a stunning turn of events for a researcher of his stature. That says a lot about how worthless the whole science journal and peer review process is. At this rate your assumption is that 50% of what is published is complete junk.

wolverine876 3y ago
That would be a far less scientific, and true, conclusion than what was in the papers. From the article: > Retractions remain exceedingly rare for scientific papers: Just eight out of every 10,000 are retracted, according to a Retraction Watch database.
- teh_infallible 3y ago
  If retractions are exceedingly rare, it could imply that, once published, fraudulent papers have a tendency to persist.
  - wolverine876 3y ago
    Yes, it could imply lots of things. To focus on only that possible implication shows a bias; to focus on any possible implication without data shows a disinterest in actual science.
  - bsder 3y ago
    It's also a recent thing that both wide dissemination and the "reproducibility crisis" have gained momentum. It has become a LOT easier for non-experts to hunt down fraud as electronic dissemination has widened.
- jyxent 3y ago
  You misunderstand. The issue is that many more papers should be retracted. A paper this major that couldn't be replicated still took another decade to be retracted. There are many more papers that are less cited that will never get looked at in enough detail to discover fraud.
  - wolverine876 3y ago
    You're just making up results, sans data. Your paper won't even pass peer review - on HN.
    - jyxent 3y ago
      And you downplay evidence of fraud. You mentioned studies where 1/3 to 1/2 of reviewed papers are found to not be replicable and say they get similar but weaker results. This is evidence of p-hacking or data manipulation. You should expect far less papers to not be replicable given proper statistics.
      - Beldin 3y ago
        First: your arguments, even if I accept them as-is, do not constitute evidence of fraud, but of a systemic problem. Second: your argumentation raises the question: what are reasons for retraction? Shoddy science is (currently) not a reason for retraction. So just because a paper's results aren't replicable doesn't mean it should be retracted. Deliberate falsification is. But you'd need to distinguish those from not-perfect science. Last, a sanity check: your comment implies/suggests that 1/3 to 1/2 of published science is fraudulent because it's not replicable. But fraudsters might accidentally hit upon the truth - at least sometimes. Moreover, plenty of research answers a binary question (yes/no, which of the two has a greater effect, etc). For those research questions, just guessing the answer gives you a50% chance of being right. All of that implies that, following your logic, a huge portion of published research is based on fraudulent practices - not mistakes, but deliberately falsified contents. Add to that the "meta-fraud" (playing with authorship, citations), and one has to wonder if every second scientist on the planet is cheating and faking everything they publish. To me, that is an obvious sign that the reasoning went off rails somewhere - much like the result of a sine function being outside of [-1, 1]. So no, you shouldn't expect 1/3 to 1/2 of all published research to require retraction.
  - arp242 3y ago
    > The issue is that many more papers should be retracted. The reality is we don't know. Should it be more than 8 out of 10,000? Almost certainly, since quite a lot of papers barely get any attention after publication. But should the number be 10 out of 10,000? 20 out of 10,000? 100 out of 10,000? We don't know. And boring papers without any interesting conclusions published just for CV reasons also don't really matter, and the 10,000 figure is also wrong, and we also don't really know what the correct value for that figure is.
- vlovich123 3y ago
  All a small retraction rate tells us is that it’s very difficult to get a retraction. But as with peer review, whether or not there’s a lot of retractions tells you little about the quality of the unretracted papers. Think about how much attention has to be paid for retractions to happen, how long the work existed without retraction, how difficult it was to get a retraction for outright fraud (ignoring work that itself was poor quality). This maybe says more about the incentives against retraction which then explains the low retraction counts better than whether or not the remaining papers are good. After all, the papers that have been retracted are high citation count papers that are foundational for certain avenues of research (this one for Alzheimer’s, the other famous fraudulent work about honesty pledges, etc). What does this say about all the papers that then cite these papers as work they’re basing theirs off of. What about the papers that then “agree” with papers that turn out to be completely fraudulent?
  - wolverine876 3y ago
    > All a small retraction rate tells us is that it’s very difficult to get a retraction. You're just making the data fit your preconceived beliefs, which would is how we get fraudulant research.
    - ekianjo 3y ago
      > making the data fit What? this is literally what happened (someone pushing hard to challenge the results) for these 4 retractions related to this guy's work. You don't get a retraction just by posting a comment on HN or something.
    - vlovich123 3y ago
      I don’t think that’s correct. I’m saying that the retraction rate tells us nothing either way and highlighting why a low retraction rate could exist even in the face of massive problems using actual facts that are not in question as examples to illustrate that point. You’re the one making definitive statements of a position and then claiming the data backs up your position and then arguing with anyone who posits that your interpretation of the data may be flawed.
    - 3y ago
- ekianjo 3y ago
  Erm... > Retractions remain exceedingly rare for scientific papers This retraction actually happened because someone BOTHERED to check the results and push for hard questions. You can bet that this does not happen for every publication... So...
- wolverine876 3y ago
  More data: > [Elisabeth] Bik later told The [Stanford] Daily in an interview, relying on her experience of investigating more than 20,000 papers for research misconduct and causing nearly 1,000 retractions and roughly the same number of corrections to make that claim. https://stanforddaily.com/2022/11/29/stanford-presidents-research-under-investigation-for-scientific-misconduct-university-admits-mistakes/ https://stanforddaily.com/2022/11/29/stanford-presidents-res... Let's assume Bik chooses papers to investigate because, to a significant degree, her expert judgment is that they are most likely to have fraud, and Bik may know that topic better than anyone in the world. Of the 20K most likely candidates, 5% were retracted and 5% corrected, and 90% apparently needed neither (though possibly they were inconclusive or she couldn't convince publications or authors to retract).
fnordpiglet 3y ago
Given how difficult it is to know what’s true in some of these fields and what’s fiction, that might be a pretty good rate. I think the bigger question is why is fraud and sloppy work so important? Maybe the incentive structure of academia is the actual problem and the peer review system is an imperfect, but perhaps effective, control? What’s the alternative to peer review?
- ekianjo 3y ago
  > Maybe the incentive structure of academia is the actual problem it's fairly clear that it is the case. You are rewarded with tenure for pushing lots of papers, regardless of their validity, since nobody actually cares about replicating anything. > peer review system is an imperfect, but perhaps effective The peer review system is just about checking points of details or calculation in a study, they never bother to actually check if the original results are valid. It's a mostly useless process that does nothing but remove the most obvious problems in the writing stage. > the alternative just like we did for hundreds of years now, proper replication baked in, by different people in different places, following the exact same conditions or protocol.
  - fnordpiglet 3y ago
    Some (many) experiments the replication is difficult it exceptionally expensive. Particularly for experiments with large exotic equipment, huge populations, etc. But certainly replication should be incentivized - sadly I think even then it’ll be largely a secondary behavior.
- mike_hearn 3y ago
  > Maybe the incentive structure of academia is the actual problem There are definitely major problems in academic research (the number of retracted papers should be much higher than it is), but it's worth noting that Tessier-Lavigne did the non-reproducible research whilst working at Genentech. One of the strangest parts of this story is how Genentech apparently already knew that the claims didn't replicate, but allowed MTL to publish a paper in Nature about it anyway ... and then went ahead and presented the claims to investors. MTL left soon after Genentech was acquired, and when they concluded years later that the paper was definitely bad they had apparently lost the ability to retract it, even though it had been done by their own employees. Whilst papers are not much of anything, presenting research to a potential acquirer that you knew wasn't reliable sounds like the basis of potential legal claims. Normally companies are very careful about that because the liability from misleading investors or acquirers can be huge.

wolverine876 3y ago

That would be a far less scientific, and true, conclusion than what was in the papers. From the article: > Retractions remain exceedingly rare for scientific papers: Just eight out of every 10,000 are retracted, according to a Retraction Watch database.

teh_infallible 3y ago
If retractions are exceedingly rare, it could imply that, once published, fraudulent papers have a tendency to persist.
- wolverine876 3y ago
  Yes, it could imply lots of things. To focus on only that possible implication shows a bias; to focus on any possible implication without data shows a disinterest in actual science.
- bsder 3y ago
  It's also a recent thing that both wide dissemination and the "reproducibility crisis" have gained momentum. It has become a LOT easier for non-experts to hunt down fraud as electronic dissemination has widened.
jyxent 3y ago
You misunderstand. The issue is that many more papers should be retracted. A paper this major that couldn't be replicated still took another decade to be retracted. There are many more papers that are less cited that will never get looked at in enough detail to discover fraud.
- wolverine876 3y ago
  You're just making up results, sans data. Your paper won't even pass peer review - on HN.
  - jyxent 3y ago
    And you downplay evidence of fraud. You mentioned studies where 1/3 to 1/2 of reviewed papers are found to not be replicable and say they get similar but weaker results. This is evidence of p-hacking or data manipulation. You should expect far less papers to not be replicable given proper statistics.
    - Beldin 3y ago
      First: your arguments, even if I accept them as-is, do not constitute evidence of fraud, but of a systemic problem. Second: your argumentation raises the question: what are reasons for retraction? Shoddy science is (currently) not a reason for retraction. So just because a paper's results aren't replicable doesn't mean it should be retracted. Deliberate falsification is. But you'd need to distinguish those from not-perfect science. Last, a sanity check: your comment implies/suggests that 1/3 to 1/2 of published science is fraudulent because it's not replicable. But fraudsters might accidentally hit upon the truth - at least sometimes. Moreover, plenty of research answers a binary question (yes/no, which of the two has a greater effect, etc). For those research questions, just guessing the answer gives you a50% chance of being right. All of that implies that, following your logic, a huge portion of published research is based on fraudulent practices - not mistakes, but deliberately falsified contents. Add to that the "meta-fraud" (playing with authorship, citations), and one has to wonder if every second scientist on the planet is cheating and faking everything they publish. To me, that is an obvious sign that the reasoning went off rails somewhere - much like the result of a sine function being outside of [-1, 1]. So no, you shouldn't expect 1/3 to 1/2 of all published research to require retraction.
- arp242 3y ago
  > The issue is that many more papers should be retracted. The reality is we don't know. Should it be more than 8 out of 10,000? Almost certainly, since quite a lot of papers barely get any attention after publication. But should the number be 10 out of 10,000? 20 out of 10,000? 100 out of 10,000? We don't know. And boring papers without any interesting conclusions published just for CV reasons also don't really matter, and the 10,000 figure is also wrong, and we also don't really know what the correct value for that figure is.
vlovich123 3y ago
All a small retraction rate tells us is that it’s very difficult to get a retraction. But as with peer review, whether or not there’s a lot of retractions tells you little about the quality of the unretracted papers. Think about how much attention has to be paid for retractions to happen, how long the work existed without retraction, how difficult it was to get a retraction for outright fraud (ignoring work that itself was poor quality). This maybe says more about the incentives against retraction which then explains the low retraction counts better than whether or not the remaining papers are good. After all, the papers that have been retracted are high citation count papers that are foundational for certain avenues of research (this one for Alzheimer’s, the other famous fraudulent work about honesty pledges, etc). What does this say about all the papers that then cite these papers as work they’re basing theirs off of. What about the papers that then “agree” with papers that turn out to be completely fraudulent?
- wolverine876 3y ago
  > All a small retraction rate tells us is that it’s very difficult to get a retraction. You're just making the data fit your preconceived beliefs, which would is how we get fraudulant research.
  - ekianjo 3y ago
    > making the data fit What? this is literally what happened (someone pushing hard to challenge the results) for these 4 retractions related to this guy's work. You don't get a retraction just by posting a comment on HN or something.
  - vlovich123 3y ago
    I don’t think that’s correct. I’m saying that the retraction rate tells us nothing either way and highlighting why a low retraction rate could exist even in the face of massive problems using actual facts that are not in question as examples to illustrate that point. You’re the one making definitive statements of a position and then claiming the data backs up your position and then arguing with anyone who posits that your interpretation of the data may be flawed.
  - 3y ago
ekianjo 3y ago
Erm... > Retractions remain exceedingly rare for scientific papers This retraction actually happened because someone BOTHERED to check the results and push for hard questions. You can bet that this does not happen for every publication... So...
wolverine876 3y ago
More data: > [Elisabeth] Bik later told The [Stanford] Daily in an interview, relying on her experience of investigating more than 20,000 papers for research misconduct and causing nearly 1,000 retractions and roughly the same number of corrections to make that claim. https://stanforddaily.com/2022/11/29/stanford-presidents-research-under-investigation-for-scientific-misconduct-university-admits-mistakes/ https://stanforddaily.com/2022/11/29/stanford-presidents-res... Let's assume Bik chooses papers to investigate because, to a significant degree, her expert judgment is that they are most likely to have fraud, and Bik may know that topic better than anyone in the world. Of the 20K most likely candidates, 5% were retracted and 5% corrected, and 90% apparently needed neither (though possibly they were inconclusive or she couldn't convince publications or authors to retract).

fnordpiglet 3y ago

Given how difficult it is to know what’s true in some of these fields and what’s fiction, that might be a pretty good rate. I think the bigger question is why is fraud and sloppy work so important? Maybe the incentive structure of academia is the actual problem and the peer review system is an imperfect, but perhaps effective, control? What’s the alternative to peer review?

ekianjo 3y ago
> Maybe the incentive structure of academia is the actual problem it's fairly clear that it is the case. You are rewarded with tenure for pushing lots of papers, regardless of their validity, since nobody actually cares about replicating anything. > peer review system is an imperfect, but perhaps effective The peer review system is just about checking points of details or calculation in a study, they never bother to actually check if the original results are valid. It's a mostly useless process that does nothing but remove the most obvious problems in the writing stage. > the alternative just like we did for hundreds of years now, proper replication baked in, by different people in different places, following the exact same conditions or protocol.
- fnordpiglet 3y ago
  Some (many) experiments the replication is difficult it exceptionally expensive. Particularly for experiments with large exotic equipment, huge populations, etc. But certainly replication should be incentivized - sadly I think even then it’ll be largely a secondary behavior.
mike_hearn 3y ago
> Maybe the incentive structure of academia is the actual problem There are definitely major problems in academic research (the number of retracted papers should be much higher than it is), but it's worth noting that Tessier-Lavigne did the non-reproducible research whilst working at Genentech. One of the strangest parts of this story is how Genentech apparently already knew that the claims didn't replicate, but allowed MTL to publish a paper in Nature about it anyway ... and then went ahead and presented the claims to investors. MTL left soon after Genentech was acquired, and when they concluded years later that the paper was definitely bad they had apparently lost the ability to retract it, even though it had been done by their own employees. Whilst papers are not much of anything, presenting research to a potential acquirer that you knew wasn't reliable sounds like the basis of potential legal claims. Normally companies are very careful about that because the liability from misleading investors or acquirers can be huge.