5 ms·

Bummer that they didn't catch it until years after it was published, but it is heartwarming to hear that the process went smoothly at least. I had a near miss

Bummer that they didn't catch it until years after it was published, but it is heartwarming to hear that the process went smoothly at least. I had a near miss publishing a paper in grad school, where buried in the ~10k LOC data analysis script there was a bug in my data processing pipeline. In summary, I had meant to evaluate B=sqrt(f(A^2)) but what I actually evaluated was B=sqrt(f(A)^2), which caused the resulting output to slightly off. In the review process, one of the reviewers looked at the output, and said wait a second this seems fishy, can you explain why it has such and such artifact. Their comments quickly allowed me to pinpoint what was going wrong, and correct the analysis script appropriately -- which actually ended up improving the result significantly! What I take away from this all is that for every article about academic misconduct and p hacking there are 100 more where the peer review process (both before and during submission to a journal) caught the issues in time. But... also probably a decent number where the errors is still in the wild to this day...

typomatic 6y ago
> What I take away from this all is that for every article about academic misconduct and p hacking there are 100 more where the peer review process (both before and during submission to a journal) caught the issues in time. p-hacking isn't a mathematical error. It cannot be "caught" because it is not presented to referees--you slightly modify your hypotheses after you experiment based on results, or you throw out "outliers" that blow up your theory. These are things that don't even show up in a paper, they happen during the compilation of the paper. How you could conclude that p-hacking is rare based on a completely unrelated experience is beyond me.
- osamagirl69 6y ago
  >How you could conclude that p-hacking is rare based on a completely unrelated experience is beyond me. I was involved in the submission of >100 papers through peer review processes, none of which involved p-hacking. In fact, they couldn't have been p-hacked because their novelty did not rely on any statistical analysis, or they were preregistered with the journal. I did have a run-in with 1 publication that I suspected involved academic misconduct (fabrication of experimental results), but it was a thesis so it did not go through the peer review process.
  - jononor 6y ago
    That is great. Pre-registration is far from standard in many fields though? And more and more papers are involving some level of statistical analysis?

typomatic 6y ago

> What I take away from this all is that for every article about academic misconduct and p hacking there are 100 more where the peer review process (both before and during submission to a journal) caught the issues in time. p-hacking isn't a mathematical error. It cannot be "caught" because it is not presented to referees--you slightly modify your hypotheses after you experiment based on results, or you throw out "outliers" that blow up your theory. These are things that don't even show up in a paper, they happen during the compilation of the paper. How you could conclude that p-hacking is rare based on a completely unrelated experience is beyond me.

osamagirl69 6y ago
>How you could conclude that p-hacking is rare based on a completely unrelated experience is beyond me. I was involved in the submission of >100 papers through peer review processes, none of which involved p-hacking. In fact, they couldn't have been p-hacked because their novelty did not rely on any statistical analysis, or they were preregistered with the journal. I did have a run-in with 1 publication that I suspected involved academic misconduct (fabrication of experimental results), but it was a thesis so it did not go through the peer review process.
- jononor 6y ago
  That is great. Pre-registration is far from standard in many fields though? And more and more papers are involving some level of statistical analysis?