4 ms·

How amazing is it to read all this on nature itself, the journal whose peer-review method failed again and again.

by fifnir 3y ago

semi-extrinsic 3y ago
Fun fact: Nature used to publish papers without any peer review whatsoever right up to the mid-60s. The selection was based on what the editors thought was interesting or provocative. Legend has it that the submissions tracking system consisted of a particularly wide windowsill, in which the submitted manuscripts were stacked up in piles for each month.
pfisherman 3y ago
Nature and Science are kind like the tabloids of scientific publishing. Legitimate news mixed with unsubstantiated gossip; all with a high “sexiness” factor that draws attention and moves copy.
dsign 3y ago
Exactly. It's bad motivations all the way: - Nature published a paper twice after the reviewers voted "no". Of course, it is not a voting process, the editor makes their decision. But evidently the editor didn't use the referee's input for that decision. That input, must I say, is free work that academics do for the journal. Click-baitiness simply weighted more to the editor and her supervisors. People should just stop paying attention to what Nature publishes. - The students... I think this is the worst part of the story. Sometimes, students are on a student visa that depends on them holding for dear life to their university position. Been there, done that. I remember when I did my PhD: four to five international students--very well prepared I must say--to one from the host country. The ones from the host country were the ones with the best soft skills but they rarely put the long hours. I'll never forget that Iranian student who was a political refugee passing as a PhD student, because she couldn't bear the indignity of an asylum process. She couldn't return to her home country either. - The replication...For the first retraction, I could find nowhere in the article that replication attempts were made. If those replication attempts were impossible to produce with the data published in the article itself, then its not a scientific paper, but a promotional brochure. In fact, some promotional materials are more extensive than Nature's articles; they really want to make the thing very short and academics are forced to jump through hoops to condense their publications, until they are practically impossible to understand, not to mention replicate. Again, the only solution to that problem is to ignore Nature.
- araes 3y ago
  The people writing the Nature article did not seem to notice the issue. The third part especially was what was occurring to me while reading. The LK-99 paper, lots of attempts at replicating the results. Crazy quick. Like front-running the published results. These results, like people never even had the thought to try, and it devolved into arguing sampling issues rather than just trying the material. There is some certain sub-segment of humanity that often feels that way. Like 8B humans will all bend over backward to let them get away with almost anything. Totally abandon all normal procedures, safety checks, laws, replication, validation, ...
- londons_explore 3y ago
  Nature should simply require the main article be short and snappy, but then put all the details needed for a replication in an online-only appendix.
autoexec 3y ago
For all the problems that Nature (and science in general) has, and while I'm sure this incident will be used by some as "evidence" that we shouldn't trust in science, ultimately this incident was another win for science. It should have happened much much sooner, but in the end, the truth was found and the record corrected. That's exactly the result we want. I appreciate that Nature's news team is willing to publish information about Nature's massively embarrassing failure to do their job reviewing the paper, and I hope they mean it when they say “We are looking at this case carefully to see what lessons can be learnt for the future.”
- dudeinjapan 3y ago
  It's a "win" for science in the sense that the truth was finally revealed. It's "loss" in the sense that our truth determining apparatus (e.g. peer review) appears to be highly unreliable, and a massive, unquantified amount of bullshit research has already passed it and is actively being used as a foundation for subsequent research.
  - dekhn 3y ago
    Realistically, we can't reduce false positives to absolutely zero without giving up on some True Positives (sensitivity/specificity tradeoff). Nature intentionally plays loose with potentially world-changing discoveries. It probably makes sense to have some journals that publish with fairly high rates of (post-hoc determined) false positives, simply to move the fields forward more quickly.
    - dudeinjapan 3y ago
      Extraordinary claims require extraordinary evidence (and validation!)
  - trogdor 3y ago
    Peer review is supposed to review the quality of the research. It is not supposed to validate the findings, let alone detect fraud.
kergonath 3y ago
It’s not the same people. As they wrote in the article, journalists are not involved in peer-review.
StressedDev 3y ago
Peer review is not a magic thing which can detect all mistakes and prevent all fraud. The biggest problem with peer review is people assume it means published research is correct. I do not think that peer review can do that. I think peer review at best gives good feedback to honest researchers and catches some mistakes. Here are my reasons why I think peer review will not reach a higher standard. 1) Peer review is hard work and probably tedious. In order to do a really good job, someone would have to spend tens or hundreds of hours reviewing a paper. The best reviewers would actually replicate the research. This probably will not happen often. 2) Peer review is not rewarded. No one is going to get tenure for being an excellent reviewer. 3) Good reviewers may even be punished. Unfortunately, many humans do not like critical feedback and hate having their work invalidated. Some of these people take revenge, and "shoot the messenger". I think the solution is to stop thinking something is correct because it is published in a peer reviewed journal. Instead, people need to recognize that while the scientific process may eventually get the right result, it often makes mistakes, and is not perfect.
dgellow 3y ago
The linked article is on Nature news. It’s a different team from Nature scientific articles. It’s mentioned in the linked article and they do some investigation in Nature’s own failures.

How amazing is it to read all this on nature itself, the journal whose peer-review method failed again and again.

by fifnir 3y ago

How amazing is it to read all this on nature itself, the journal whose peer-review method failed again and again.

Fun fact: Nature used to publish papers without any peer review whatsoever right up to the mid-60s. The selection was based on what the editors thought was interesting or provocative. Legend has it that the submissions tracking system consisted of a particularly wide windowsill, in which the submitted manuscripts were stacked up in piles for each month.

pfisherman 3y ago

Nature and Science are kind like the tabloids of scientific publishing. Legitimate news mixed with unsubstantiated gossip; all with a high “sexiness” factor that draws attention and moves copy.

dsign 3y ago

Exactly. It's bad motivations all the way: - Nature published a paper twice after the reviewers voted "no". Of course, it is not a voting process, the editor makes their decision. But evidently the editor didn't use the referee's input for that decision. That input, must I say, is free work that academics do for the journal. Click-baitiness simply weighted more to the editor and her supervisors. People should just stop paying attention to what Nature publishes. - The students... I think this is the worst part of the story. Sometimes, students are on a student visa that depends on them holding for dear life to their university position. Been there, done that. I remember when I did my PhD: four to five international students--very well prepared I must say--to one from the host country. The ones from the host country were the ones with the best soft skills but they rarely put the long hours. I'll never forget that Iranian student who was a political refugee passing as a PhD student, because she couldn't bear the indignity of an asylum process. She couldn't return to her home country either. - The replication...For the first retraction, I could find nowhere in the article that replication attempts were made. If those replication attempts were impossible to produce with the data published in the article itself, then its not a scientific paper, but a promotional brochure. In fact, some promotional materials are more extensive than Nature's articles; they really want to make the thing very short and academics are forced to jump through hoops to condense their publications, until they are practically impossible to understand, not to mention replicate. Again, the only solution to that problem is to ignore Nature.

araes 3y ago
The people writing the Nature article did not seem to notice the issue. The third part especially was what was occurring to me while reading. The LK-99 paper, lots of attempts at replicating the results. Crazy quick. Like front-running the published results. These results, like people never even had the thought to try, and it devolved into arguing sampling issues rather than just trying the material. There is some certain sub-segment of humanity that often feels that way. Like 8B humans will all bend over backward to let them get away with almost anything. Totally abandon all normal procedures, safety checks, laws, replication, validation, ...
londons_explore 3y ago
Nature should simply require the main article be short and snappy, but then put all the details needed for a replication in an online-only appendix.

autoexec 3y ago

For all the problems that Nature (and science in general) has, and while I'm sure this incident will be used by some as "evidence" that we shouldn't trust in science, ultimately this incident was another win for science. It should have happened much much sooner, but in the end, the truth was found and the record corrected. That's exactly the result we want. I appreciate that Nature's news team is willing to publish information about Nature's massively embarrassing failure to do their job reviewing the paper, and I hope they mean it when they say “We are looking at this case carefully to see what lessons can be learnt for the future.”

dudeinjapan 3y ago
It's a "win" for science in the sense that the truth was finally revealed. It's "loss" in the sense that our truth determining apparatus (e.g. peer review) appears to be highly unreliable, and a massive, unquantified amount of bullshit research has already passed it and is actively being used as a foundation for subsequent research.
- dekhn 3y ago
  Realistically, we can't reduce false positives to absolutely zero without giving up on some True Positives (sensitivity/specificity tradeoff). Nature intentionally plays loose with potentially world-changing discoveries. It probably makes sense to have some journals that publish with fairly high rates of (post-hoc determined) false positives, simply to move the fields forward more quickly.
  - dudeinjapan 3y ago
    Extraordinary claims require extraordinary evidence (and validation!)
- trogdor 3y ago
  Peer review is supposed to review the quality of the research. It is not supposed to validate the findings, let alone detect fraud.

kergonath 3y ago

It’s not the same people. As they wrote in the article, journalists are not involved in peer-review.

StressedDev 3y ago

Peer review is not a magic thing which can detect all mistakes and prevent all fraud. The biggest problem with peer review is people assume it means published research is correct. I do not think that peer review can do that. I think peer review at best gives good feedback to honest researchers and catches some mistakes. Here are my reasons why I think peer review will not reach a higher standard. 1) Peer review is hard work and probably tedious. In order to do a really good job, someone would have to spend tens or hundreds of hours reviewing a paper. The best reviewers would actually replicate the research. This probably will not happen often. 2) Peer review is not rewarded. No one is going to get tenure for being an excellent reviewer. 3) Good reviewers may even be punished. Unfortunately, many humans do not like critical feedback and hate having their work invalidated. Some of these people take revenge, and "shoot the messenger". I think the solution is to stop thinking something is correct because it is published in a peer reviewed journal. Instead, people need to recognize that while the scientific process may eventually get the right result, it often makes mistakes, and is not perfect.

dgellow 3y ago

The linked article is on Nature news. It’s a different team from Nature scientific articles. It’s mentioned in the linked article and they do some investigation in Nature’s own failures.