12 ms·

Bizarre. How do people make such big, splashy findings that can mess with people’s sense of optimism about science and innovation, without doing the simplest ty

Bizarre. How do people make such big, splashy findings that can mess with people’s sense of optimism about science and innovation, without doing the simplest types of checks on their data and methodology.

brookst 3y ago
It’s enough to make you lose faith in science.
- masklinn 3y ago
  As opposed to? The original claim being asspulled then not reviewed at all because that would be science?
  - dingnuts 3y ago
    the GP I assume means science as a monolithic institution, as opposed to the pure idea of the scientific process in isolation kind of like the difference between "trusting science" and "trusting THE science" if I had to hazard a guess presumably he doesn't mean as opposed to "traditional ways of knowing"
- neerajk 3y ago
  it is science that discovers errors in science :)
  - rrix2 3y ago
    And here we are, discovering errors!
    - BlueTemplar 3y ago
      > Now these points of data make a beautiful line. And we're out of beta. We're releasing on time.
masklinn 3y ago
Confirmation bias? It’s easy to run with the assumption you had at the back of your mind when your experiment seems to confirm it. I have definitely done that with benchmarks / profiles. It’s probably even easier when the incentives encourage “the find”.
- SubiculumCode 3y ago
  They checked their result multiple ways. The missed a bug. Its not like computer bugs never happen or anything
  - light_hue_1 3y ago
    The real question is, what will they do now? Will they own up to it and retract their broken paper that's eroding people's confidence in funding science at the highest levels? This has been an incredibly widely read and influential paper already. I have my doubts that the authors will accept that their paper is bogus. Particularly because the lead author landed a faculty position at a good institution based exclusively on this junk paper.
    - SubiculumCode 3y ago
      umm. probably.
SubiculumCode 3y ago
Trash comment. 1st. Splashy often comes from the media, not the scientists 2nd. One of the ways we discover problems with data is by plotting. When the plot library has a bug that hides a problem, well shit. 3rd. They did check their own findings multiple ways. Mistakes happen. The biggest critics of scientific mistakes are often those that have never done science themselves. Its easy, and its a cheap play.
- keenmaster 3y ago
  Genuine question: you don’t think that outlier analysis (with moderate diligence) would have picked up the error in the original study, despite the plotting library issue? Just look at this link - this wasn’t detectable with any type of cross-validation? https://arxiv.org/html/2402.14583v1/x1.png https://arxiv.org/html/2402.14583v1/x1.png
janosdebugs 3y ago
No, the question is: how did peer review not catch it? I have the impression that reviewers don't have the time or incentive to give papers more that a cursory review. Independent of this case, a great many papers are published where the only "proof" is a user study or survey with an extremely low number of participants, but it still gets published. Many papers don't publish their datasets and don't contain enough detail to try and replicate their results. There should be a real incentive/compensation for reviewing properly and real consequences if a paper gets retracted for reasons that should have been caught in review. In this case it's fortunate that it did get found out in the end.
- BeetleB 3y ago
  Can you describe how you would have expected peer reviewers to catch this?
  - janosdebugs 3y ago
    I am not an expert in statistics, but i have read quite a few papers in my area (IT/Kubernetes/etc) that had an obviously faulty methodology if you have experience at all. Reading the other comments, this software should have never been used in this manner, which it seems to me someone who is hopefully well-versed in this area should have caught. Then again, the reviewers may have had very little experience in this area. (This happened to me, when the review came back the reviewers admitted themselves in the form that this was the case.)
- phdelightful 3y ago
  Your impression is correct. Peer review would never catch this. Peer review basically assumes the counter party is operating in good faith, and as a result a thorough peer review basically is the following: * is the treatment of existing work semi-thorough (even experts don’t know everything) and fair? * are the claims novel w.r.t the existing work? If not, provide a reference to someone who has already done it. * can you understand the experiments? * do the experiments and their results lead to the conclusions claimed as novel? * does the writing inhibit understanding of the technical content? No peer review I have ever seen or done would catch anything but the most egregious bug of this nature.

brookst 3y ago

It’s enough to make you lose faith in science.

masklinn 3y ago
As opposed to? The original claim being asspulled then not reviewed at all because that would be science?
- dingnuts 3y ago
  the GP I assume means science as a monolithic institution, as opposed to the pure idea of the scientific process in isolation kind of like the difference between "trusting science" and "trusting THE science" if I had to hazard a guess presumably he doesn't mean as opposed to "traditional ways of knowing"
neerajk 3y ago
it is science that discovers errors in science :)
- rrix2 3y ago
  And here we are, discovering errors!
  - BlueTemplar 3y ago
    > Now these points of data make a beautiful line. And we're out of beta. We're releasing on time.

masklinn 3y ago

Confirmation bias? It’s easy to run with the assumption you had at the back of your mind when your experiment seems to confirm it. I have definitely done that with benchmarks / profiles. It’s probably even easier when the incentives encourage “the find”.

SubiculumCode 3y ago
They checked their result multiple ways. The missed a bug. Its not like computer bugs never happen or anything
- light_hue_1 3y ago
  The real question is, what will they do now? Will they own up to it and retract their broken paper that's eroding people's confidence in funding science at the highest levels? This has been an incredibly widely read and influential paper already. I have my doubts that the authors will accept that their paper is bogus. Particularly because the lead author landed a faculty position at a good institution based exclusively on this junk paper.
  - SubiculumCode 3y ago
    umm. probably.

SubiculumCode 3y ago

Trash comment. 1st. Splashy often comes from the media, not the scientists 2nd. One of the ways we discover problems with data is by plotting. When the plot library has a bug that hides a problem, well shit. 3rd. They did check their own findings multiple ways. Mistakes happen. The biggest critics of scientific mistakes are often those that have never done science themselves. Its easy, and its a cheap play.

keenmaster 3y ago
Genuine question: you don’t think that outlier analysis (with moderate diligence) would have picked up the error in the original study, despite the plotting library issue? Just look at this link - this wasn’t detectable with any type of cross-validation? https://arxiv.org/html/2402.14583v1/x1.png https://arxiv.org/html/2402.14583v1/x1.png

janosdebugs 3y ago

No, the question is: how did peer review not catch it? I have the impression that reviewers don't have the time or incentive to give papers more that a cursory review. Independent of this case, a great many papers are published where the only "proof" is a user study or survey with an extremely low number of participants, but it still gets published. Many papers don't publish their datasets and don't contain enough detail to try and replicate their results. There should be a real incentive/compensation for reviewing properly and real consequences if a paper gets retracted for reasons that should have been caught in review. In this case it's fortunate that it did get found out in the end.

BeetleB 3y ago
Can you describe how you would have expected peer reviewers to catch this?
- janosdebugs 3y ago
  I am not an expert in statistics, but i have read quite a few papers in my area (IT/Kubernetes/etc) that had an obviously faulty methodology if you have experience at all. Reading the other comments, this software should have never been used in this manner, which it seems to me someone who is hopefully well-versed in this area should have caught. Then again, the reviewers may have had very little experience in this area. (This happened to me, when the review came back the reviewers admitted themselves in the form that this was the case.)
phdelightful 3y ago
Your impression is correct. Peer review would never catch this. Peer review basically assumes the counter party is operating in good faith, and as a result a thorough peer review basically is the following: * is the treatment of existing work semi-thorough (even experts don’t know everything) and fair? * are the claims novel w.r.t the existing work? If not, provide a reference to someone who has already done it. * can you understand the experiments? * do the experiments and their results lead to the conclusions claimed as novel? * does the writing inhibit understanding of the technical content? No peer review I have ever seen or done would catch anything but the most egregious bug of this nature.