8 ms·

What my retraction taught me

128 points by mellosouls 6y ago

osamagirl69 6y ago
Bummer that they didn't catch it until years after it was published, but it is heartwarming to hear that the process went smoothly at least. I had a near miss publishing a paper in grad school, where buried in the ~10k LOC data analysis script there was a bug in my data processing pipeline. In summary, I had meant to evaluate B=sqrt(f(A^2)) but what I actually evaluated was B=sqrt(f(A)^2), which caused the resulting output to slightly off. In the review process, one of the reviewers looked at the output, and said wait a second this seems fishy, can you explain why it has such and such artifact. Their comments quickly allowed me to pinpoint what was going wrong, and correct the analysis script appropriately -- which actually ended up improving the result significantly! What I take away from this all is that for every article about academic misconduct and p hacking there are 100 more where the peer review process (both before and during submission to a journal) caught the issues in time. But... also probably a decent number where the errors is still in the wild to this day...
- typomatic 6y ago
  > What I take away from this all is that for every article about academic misconduct and p hacking there are 100 more where the peer review process (both before and during submission to a journal) caught the issues in time. p-hacking isn't a mathematical error. It cannot be "caught" because it is not presented to referees--you slightly modify your hypotheses after you experiment based on results, or you throw out "outliers" that blow up your theory. These are things that don't even show up in a paper, they happen during the compilation of the paper. How you could conclude that p-hacking is rare based on a completely unrelated experience is beyond me.
  - osamagirl69 6y ago
    >How you could conclude that p-hacking is rare based on a completely unrelated experience is beyond me. I was involved in the submission of >100 papers through peer review processes, none of which involved p-hacking. In fact, they couldn't have been p-hacked because their novelty did not rely on any statistical analysis, or they were preregistered with the journal. I did have a run-in with 1 publication that I suspected involved academic misconduct (fabrication of experimental results), but it was a thesis so it did not go through the peer review process.
    - jononor 6y ago
      That is great. Pre-registration is far from standard in many fields though? And more and more papers are involving some level of statistical analysis?
deleted 6y ago
[deleted]
einpoklum 6y ago
> That [the method of statistical analysis] could be a problem in our particular context didn’t dawn on me and my colleagues — nor on anyone else in the field — before [whoever]'s discovery. This is troubling to me. It does not seem like the bias was due to something arcane, some finer point in advanced statistics, something hidden from view etc. The poster says: > It involved regression towards the mean — when noisy data are measured repeatedly, values that at first look extreme become less so. This may not be 100% straightforward when it's buried in the middle of a paper, but if you actually consider the methodolgy you are likely to notice this happening. So here's what _I_ learn from this case: * It is possible that the reviewers at Nature don't properly scrutinize the methodological soundness of some submissions (I say "possible" since this is a single example, not a pattern) * PhD avisors, like the author's, may not be exercising due dilligence on statistical research done with their PhD candidates. The author's advisor had this to say: > "It's great that we’ve persisted in attempting to understand our methodology and findings!” so he says it's "great" that they did not fully understand their methodology before submitting a paper using it. Maybe that's not exactly what he meant, but still, pretty worrying.
- bachmeier 6y ago
  You present a view of journals as an outlet for correct results. Realistically, that's not possible, and I wish more people accepted that it's not possible. If a result is confirmed in many studies by many authors using many datasets and many methodologies, that's how we know we can trust a result. I personally do not put much weight on a single paper's results unless there's something special about it. The most important thing is that everything about the investigation is open: the methodology (including everything not found in the paper), the programs, the data. It's more like posting code for a large project on Github and then new researchers make PRs to correct bugs or extend the program in useful ways.
  - einpoklum 6y ago
    I think you're conflating correctness and degrees of validity. Yes, I expect journals to have to have correct results - not in the sense that the generalizations from their findings are universally valid, but in that the statements of fact and of logical implications are valid. To be more explicit: * "Events X, Y, Z occurred" <- expect this kind of sentences to be correct * "We did X, Y, Z" <- expect this kind of sentences to be correct * "A and A->B, so B" <- expect this kind of sentences to be correct * "We therefore conclude that X" <- Don't expect X to necessarily be correct.
smoyer 6y ago
I listen to a weekly podcast titled "Everything Hertz" in which includes a large amount of discussion related to methodology as well as problematically citing previous research etc. It's pretty fascinating (as is this article) - https://everythinghertz.com/ https://everythinghertz.com/.
- jarenmf 6y ago
  Thank you, this looks interesting indeed
sxp 6y ago
The workflow for modern scientific research always seems so amateurish from the perspective of basic software development. > We set up a video meeting, and decided that Susanne would go through simulations, and I would go through my old data, if I could dig them up. That was a challenge. Only months before, my current university had suffered a cyberattack, and access to my back-up drive was prohibited at first. ... I spent a week piecing together the necessary files and coding a pipeline to reproduce the original findings. To my horror, I also reproduced the problem that Susanne had found. The main issue was that I had used the same data for selection and comparison, a circularity that crops up again and again. The proper solution would be to publish the code along with the paper so that others could directly review it. Ideally, the raw data would also be published, but there might be privacy issues with that. Peer review for journals is supposed to catch these type of workflow problems, but just throwing the code & paper up onto GitHub would probably be more effective than publishing just the paper and waiting for others to catch the problem.
- bachmeier 6y ago
  The journal I edit has a policy that the code and data has to be made available to the reviewers. I was surprised at the number of reviewers that dig into the code.
  - semi-extrinsic 6y ago
    TBH, code (even when all you can do is just looking at it and understanding how it works) is probably the most useful thing to the reviewers themselves. Imagine the equivalent for reviewing an experimental paper, if you were given a free instant travel to the place where they did the experiment and you could look at their setup and tinker with it and see how it works. I guarantee you a lot of reviewers would go for that.
    - bachmeier 6y ago
      I've learned that what you say is true. This is particularly the case for papers that use small datasets and not especially complicated empirical analysis.
mrandish 6y ago
Really happy to see this in a major journal. Simultaneous sharing of code and data necessary to replicate a result should be the minimum expectation for journal publication.
kwdc 6y ago
This stuck out as an important point: [As a] "student, I was even told never to attempt to replicate before I publish." An actual scientist would never say this. I know about the real world, costs and pragmatism etc. No actual scientist should ever be proud to refuse any replication attempts to allow verification. Its that simple. Fundamentally that should be enough to initiate various investigation towards disciplinary proceedings. Its that serious a offence, eg a form of corruption. Fake scientists, charlatans, "science believers", science authoritarians... sure. But who wants them around? Admitting a mistake is actually good science.
kstenerud 6y ago
I shudder to imagine what the world of software would be like if discovering bugs in your code would limit your career prospects similar to how negative results or methodology errors do in scientific research.
lproven 6y ago
Fascinating. The problem is far more rife than I realised... https://www.sciencemag.org/news/2018/10/what-massive-database-retracted-papers-reveals-about-science-publishing-s-death-penalty https://www.sciencemag.org/news/2018/10/what-massive-databas... https://retractionwatch.com/2014/05/15/which-countries-have-the-most-retractions-for-which-reasons/ https://retractionwatch.com/2014/05/15/which-countries-have-...

128 points by mellosouls 6y ago

Bummer that they didn't catch it until years after it was published, but it is heartwarming to hear that the process went smoothly at least. I had a near miss publishing a paper in grad school, where buried in the ~10k LOC data analysis script there was a bug in my data processing pipeline. In summary, I had meant to evaluate B=sqrt(f(A^2)) but what I actually evaluated was B=sqrt(f(A)^2), which caused the resulting output to slightly off. In the review process, one of the reviewers looked at the output, and said wait a second this seems fishy, can you explain why it has such and such artifact. Their comments quickly allowed me to pinpoint what was going wrong, and correct the analysis script appropriately -- which actually ended up improving the result significantly! What I take away from this all is that for every article about academic misconduct and p hacking there are 100 more where the peer review process (both before and during submission to a journal) caught the issues in time. But... also probably a decent number where the errors is still in the wild to this day...

typomatic 6y ago
> What I take away from this all is that for every article about academic misconduct and p hacking there are 100 more where the peer review process (both before and during submission to a journal) caught the issues in time. p-hacking isn't a mathematical error. It cannot be "caught" because it is not presented to referees--you slightly modify your hypotheses after you experiment based on results, or you throw out "outliers" that blow up your theory. These are things that don't even show up in a paper, they happen during the compilation of the paper. How you could conclude that p-hacking is rare based on a completely unrelated experience is beyond me.
- osamagirl69 6y ago
  >How you could conclude that p-hacking is rare based on a completely unrelated experience is beyond me. I was involved in the submission of >100 papers through peer review processes, none of which involved p-hacking. In fact, they couldn't have been p-hacked because their novelty did not rely on any statistical analysis, or they were preregistered with the journal. I did have a run-in with 1 publication that I suspected involved academic misconduct (fabrication of experimental results), but it was a thesis so it did not go through the peer review process.
  - jononor 6y ago
    That is great. Pre-registration is far from standard in many fields though? And more and more papers are involving some level of statistical analysis?

deleted 6y ago

[deleted]

einpoklum 6y ago

> That [the method of statistical analysis] could be a problem in our particular context didn’t dawn on me and my colleagues — nor on anyone else in the field — before [whoever]'s discovery. This is troubling to me. It does not seem like the bias was due to something arcane, some finer point in advanced statistics, something hidden from view etc. The poster says: > It involved regression towards the mean — when noisy data are measured repeatedly, values that at first look extreme become less so. This may not be 100% straightforward when it's buried in the middle of a paper, but if you actually consider the methodolgy you are likely to notice this happening. So here's what _I_ learn from this case: * It is possible that the reviewers at Nature don't properly scrutinize the methodological soundness of some submissions (I say "possible" since this is a single example, not a pattern) * PhD avisors, like the author's, may not be exercising due dilligence on statistical research done with their PhD candidates. The author's advisor had this to say: > "It's great that we’ve persisted in attempting to understand our methodology and findings!” so he says it's "great" that they did not fully understand their methodology before submitting a paper using it. Maybe that's not exactly what he meant, but still, pretty worrying.

bachmeier 6y ago
You present a view of journals as an outlet for correct results. Realistically, that's not possible, and I wish more people accepted that it's not possible. If a result is confirmed in many studies by many authors using many datasets and many methodologies, that's how we know we can trust a result. I personally do not put much weight on a single paper's results unless there's something special about it. The most important thing is that everything about the investigation is open: the methodology (including everything not found in the paper), the programs, the data. It's more like posting code for a large project on Github and then new researchers make PRs to correct bugs or extend the program in useful ways.
- einpoklum 6y ago
  I think you're conflating correctness and degrees of validity. Yes, I expect journals to have to have correct results - not in the sense that the generalizations from their findings are universally valid, but in that the statements of fact and of logical implications are valid. To be more explicit: * "Events X, Y, Z occurred" <- expect this kind of sentences to be correct * "We did X, Y, Z" <- expect this kind of sentences to be correct * "A and A->B, so B" <- expect this kind of sentences to be correct * "We therefore conclude that X" <- Don't expect X to necessarily be correct.

smoyer 6y ago

I listen to a weekly podcast titled "Everything Hertz" in which includes a large amount of discussion related to methodology as well as problematically citing previous research etc. It's pretty fascinating (as is this article) - https://everythinghertz.com/ https://everythinghertz.com/.

jarenmf 6y ago
Thank you, this looks interesting indeed

sxp 6y ago

The workflow for modern scientific research always seems so amateurish from the perspective of basic software development. > We set up a video meeting, and decided that Susanne would go through simulations, and I would go through my old data, if I could dig them up. That was a challenge. Only months before, my current university had suffered a cyberattack, and access to my back-up drive was prohibited at first. ... I spent a week piecing together the necessary files and coding a pipeline to reproduce the original findings. To my horror, I also reproduced the problem that Susanne had found. The main issue was that I had used the same data for selection and comparison, a circularity that crops up again and again. The proper solution would be to publish the code along with the paper so that others could directly review it. Ideally, the raw data would also be published, but there might be privacy issues with that. Peer review for journals is supposed to catch these type of workflow problems, but just throwing the code & paper up onto GitHub would probably be more effective than publishing just the paper and waiting for others to catch the problem.

bachmeier 6y ago
The journal I edit has a policy that the code and data has to be made available to the reviewers. I was surprised at the number of reviewers that dig into the code.
- semi-extrinsic 6y ago
  TBH, code (even when all you can do is just looking at it and understanding how it works) is probably the most useful thing to the reviewers themselves. Imagine the equivalent for reviewing an experimental paper, if you were given a free instant travel to the place where they did the experiment and you could look at their setup and tinker with it and see how it works. I guarantee you a lot of reviewers would go for that.
  - bachmeier 6y ago
    I've learned that what you say is true. This is particularly the case for papers that use small datasets and not especially complicated empirical analysis.

mrandish 6y ago

Really happy to see this in a major journal. Simultaneous sharing of code and data necessary to replicate a result should be the minimum expectation for journal publication.

kwdc 6y ago

This stuck out as an important point: [As a] "student, I was even told never to attempt to replicate before I publish." An actual scientist would never say this. I know about the real world, costs and pragmatism etc. No actual scientist should ever be proud to refuse any replication attempts to allow verification. Its that simple. Fundamentally that should be enough to initiate various investigation towards disciplinary proceedings. Its that serious a offence, eg a form of corruption. Fake scientists, charlatans, "science believers", science authoritarians... sure. But who wants them around? Admitting a mistake is actually good science.

kstenerud 6y ago

I shudder to imagine what the world of software would be like if discovering bugs in your code would limit your career prospects similar to how negative results or methodology errors do in scientific research.

lproven 6y ago

Fascinating. The problem is far more rife than I realised... https://www.sciencemag.org/news/2018/10/what-massive-database-retracted-papers-reveals-about-science-publishing-s-death-penalty https://www.sciencemag.org/news/2018/10/what-massive-databas... https://retractionwatch.com/2014/05/15/which-countries-have-the-most-retractions-for-which-reasons/ https://retractionwatch.com/2014/05/15/which-countries-have-...