4 ms·

> Soon after the CSH paper was published, Jorge Hirsch, a condensed-matter theorist at the University of California, San Diego, began pressing Dias to release t

> Soon after the CSH paper was published, Jorge Hirsch, a condensed-matter theorist at the University of California, San Diego, began pressing Dias to release the raw magnetic-susceptibility data, which were not included in the paper. More than a year later, Dias and Salamat finally made the raw data public. Explain to me why Nature would publish a paper without having access to/requiring the data. Could it be Nature wanted to be the first to publish such a ground breaking discovery? Dishonest people are simply a fact of life. The entire point - supposedly - of peer review is to insure that scientist do not make unsubstantiated claims. Job well done Nature. Job well done.

tornato7 3y ago
I would like it noted that I had Jorge Hirsch as a professor many years ago, and his grading was extremely harsh - you needed to be flawless at showing your work, not just getting the right answer. I remember getting a 14% on one of his tests, the worst score I ever got on any test in my life. It turned out to be a B+ after the curve was applied. Hence, it doesn't surprise me at all to see that he was the one to call out Dias. Some things never change!
- deleted 3y ago
  [deleted]
0cf8612b2e1e 3y ago
Peer review is not designed to detect fraud. It is meant to improve the quality of work, by giving a publication a fresh set of eyes who might identify gaps in the original research.
- strangattractor 3y ago
  Not having data is quite a significant gap. BTW: I am not criticizing the actual reviewer of the paper. Data availability should be a policy of the journal and required for publication. How can you improve quality without access to the data? In this instance requiring data would have likely prevented the paper from even reaching peer review and wasting a busy individuals time.
  - 0cf8612b2e1e 3y ago
    That was one particular dataset being requested. There are near infinite ways to characterize a material, and even the best lab is only going to have the time, resources, and expertise to collect and present some of them. I am not in the field, so I cannot comment on the specific relevance of the request, but in my area, there are absolutely some mandatory data inclusions which should block a paper from being published. Presumably this was not on that list.
    - strangattractor 3y ago
      There are numerous ways to embargo data and make it available to reviewers before publication and to the public after publication. [1] This isn't a new idea. They simply did not do it. Nature is a "premier journal" known for publishing cutting edge research. Any data - in this case a bunch of numbers - used to support the conclusion should be made available. I am sure there are disciplines where the evidence would be difficult to provide directly - fossils, physical items etc. Maybe this PI did the experiment himself - thought he won a Noble - only to find he could not in fact reproduce the results after publication - oops. IMO Nature did not do its job. Just pointing that out. [1] https://plos.org/resource/how-to-store-and-manage-your-data/ https://plos.org/resource/how-to-store-and-manage-your-data/
  - BeetleB 3y ago
    > Not having data is quite a significant gap. Requiring (raw) data is relatively new, and is only in a few fields. For most of the history of peer review data was not required. In fact, none of the PIs I worked in during grad school would ever agree to releasing the raw data. I'm pretty sure they're still not in most of physics. IMO, it's reasonable to require it, but understand that this is a relatively new requirement and not part of peer review culture.
    - worik 3y ago
      > Requiring (raw) data is relatively new, and is only in a few fields. That on its own is scandalous I was reading of the "reproducibility crisis", in computing, in the nineties Science still wants to be the warm friendly club, whilst playing competitively for high personal stakes. Cannot be both. Fraud, deliberately cynical fraud and hopeful naive fraud, has been rampant for decades The journals benefit directly and seem to be utterly complicit. The system is dreadfully broken Nobody should wonder why people do not trust science
- pdonis 3y ago
  > Peer review is not designed to detect fraud. Maybe it should be. "Peer reviewed" is taken as a sign of reliability in many contexts, including public policy. If it isn't actually such a sign, then either it needs to change so it is, or we need to stop taking it as a sign of something it isn't.
  - 0cf8612b2e1e 3y ago
    I am not sure you appreciate the scope of distrusting submissions. Does this mean the reviewer has to clean-slate replicate everything? What if the author spent three years breeding a particular mouse model? Purified 8pg of an antibody extracted from flea tears? Spent six months of HPC time to simulate molecular dynamics? Some science is hard and incredibly difficult and expensive to do. Even just reevaluating a submitted dataset is no guarantee. Someone committing an elaborate fraud could have faked the raw datasets as well.
    - pdonis 3y ago
      > I am not sure you appreciate the scope of distrusting submissions. What you're saying is, the first option I gave, to fix the peer review process so it is a good indicator of the reliability of research, is not feasible. If that's the case, that just means we have to pick the second option: stop treating peer reviewed research as if it is reliable.
      - beowulfey 3y ago
        That's actually exactly correct. Peer review is supposed to characterize the research in terms of quality, not validate it. Otherwise, those peers who validate it would practically be co-authors. It's a system built upon honesty. A dishonest person is obviously going to take advantage of it. Really blatant things are caught in peer review, but sneaky fraud will definitely slip through.
        pdonis 3y ago
        > Peer review is supposed to characterize the research in terms of quality Fraudulent research would obviously not be of the desired quality, wouldn't it? > It's a system built upon honesty. But if the purpose is to characterize the research in terms of quality, this can't be right. Since fraud is known to happen, and there are known to be incentives for fraud, any process that claims to judge the quality of research would have to have some way of checking for fraud.
deleted 3y ago
[deleted]
dekhn 3y ago
Nature very specifically wants to be the venue where the most important and impactful discoveries are first reported, and they tolerate a very high false positive rate to achieve it. Quick, light peer review is part of that process. We always joked that if you read it in Nature, it was certain to be wrong, but realistically, it meant that the topic was likely an important one with a small number of competitive groups who could all evaluate each other's papers and will loudly complain about every inconsistency or irreproducible detail.

tornato7 3y ago

I would like it noted that I had Jorge Hirsch as a professor many years ago, and his grading was extremely harsh - you needed to be flawless at showing your work, not just getting the right answer. I remember getting a 14% on one of his tests, the worst score I ever got on any test in my life. It turned out to be a B+ after the curve was applied. Hence, it doesn't surprise me at all to see that he was the one to call out Dias. Some things never change!

deleted 3y ago
[deleted]

0cf8612b2e1e 3y ago

Peer review is not designed to detect fraud. It is meant to improve the quality of work, by giving a publication a fresh set of eyes who might identify gaps in the original research.

strangattractor 3y ago
Not having data is quite a significant gap. BTW: I am not criticizing the actual reviewer of the paper. Data availability should be a policy of the journal and required for publication. How can you improve quality without access to the data? In this instance requiring data would have likely prevented the paper from even reaching peer review and wasting a busy individuals time.
- 0cf8612b2e1e 3y ago
  That was one particular dataset being requested. There are near infinite ways to characterize a material, and even the best lab is only going to have the time, resources, and expertise to collect and present some of them. I am not in the field, so I cannot comment on the specific relevance of the request, but in my area, there are absolutely some mandatory data inclusions which should block a paper from being published. Presumably this was not on that list.
  - strangattractor 3y ago
    There are numerous ways to embargo data and make it available to reviewers before publication and to the public after publication. [1] This isn't a new idea. They simply did not do it. Nature is a "premier journal" known for publishing cutting edge research. Any data - in this case a bunch of numbers - used to support the conclusion should be made available. I am sure there are disciplines where the evidence would be difficult to provide directly - fossils, physical items etc. Maybe this PI did the experiment himself - thought he won a Noble - only to find he could not in fact reproduce the results after publication - oops. IMO Nature did not do its job. Just pointing that out. [1] https://plos.org/resource/how-to-store-and-manage-your-data/ https://plos.org/resource/how-to-store-and-manage-your-data/
- BeetleB 3y ago
  > Not having data is quite a significant gap. Requiring (raw) data is relatively new, and is only in a few fields. For most of the history of peer review data was not required. In fact, none of the PIs I worked in during grad school would ever agree to releasing the raw data. I'm pretty sure they're still not in most of physics. IMO, it's reasonable to require it, but understand that this is a relatively new requirement and not part of peer review culture.
  - worik 3y ago
    > Requiring (raw) data is relatively new, and is only in a few fields. That on its own is scandalous I was reading of the "reproducibility crisis", in computing, in the nineties Science still wants to be the warm friendly club, whilst playing competitively for high personal stakes. Cannot be both. Fraud, deliberately cynical fraud and hopeful naive fraud, has been rampant for decades The journals benefit directly and seem to be utterly complicit. The system is dreadfully broken Nobody should wonder why people do not trust science
pdonis 3y ago
> Peer review is not designed to detect fraud. Maybe it should be. "Peer reviewed" is taken as a sign of reliability in many contexts, including public policy. If it isn't actually such a sign, then either it needs to change so it is, or we need to stop taking it as a sign of something it isn't.
- 0cf8612b2e1e 3y ago
  I am not sure you appreciate the scope of distrusting submissions. Does this mean the reviewer has to clean-slate replicate everything? What if the author spent three years breeding a particular mouse model? Purified 8pg of an antibody extracted from flea tears? Spent six months of HPC time to simulate molecular dynamics? Some science is hard and incredibly difficult and expensive to do. Even just reevaluating a submitted dataset is no guarantee. Someone committing an elaborate fraud could have faked the raw datasets as well.
  - pdonis 3y ago
    > I am not sure you appreciate the scope of distrusting submissions. What you're saying is, the first option I gave, to fix the peer review process so it is a good indicator of the reliability of research, is not feasible. If that's the case, that just means we have to pick the second option: stop treating peer reviewed research as if it is reliable.
    - beowulfey 3y ago
      That's actually exactly correct. Peer review is supposed to characterize the research in terms of quality, not validate it. Otherwise, those peers who validate it would practically be co-authors. It's a system built upon honesty. A dishonest person is obviously going to take advantage of it. Really blatant things are caught in peer review, but sneaky fraud will definitely slip through.
      - pdonis 3y ago
        > Peer review is supposed to characterize the research in terms of quality Fraudulent research would obviously not be of the desired quality, wouldn't it? > It's a system built upon honesty. But if the purpose is to characterize the research in terms of quality, this can't be right. Since fraud is known to happen, and there are known to be incentives for fraud, any process that claims to judge the quality of research would have to have some way of checking for fraud.

deleted 3y ago

[deleted]

dekhn 3y ago

Nature very specifically wants to be the venue where the most important and impactful discoveries are first reported, and they tolerate a very high false positive rate to achieve it. Quick, light peer review is part of that process. We always joked that if you read it in Nature, it was certain to be wrong, but realistically, it meant that the topic was likely an important one with a small number of competitive groups who could all evaluate each other's papers and will loudly complain about every inconsistency or irreproducible detail.