5 ms·

Not having data is quite a significant gap. BTW: I am not criticizing the actual reviewer of the paper. Data availability should be a policy of the journal and

Not having data is quite a significant gap. BTW: I am not criticizing the actual reviewer of the paper. Data availability should be a policy of the journal and required for publication. How can you improve quality without access to the data? In this instance requiring data would have likely prevented the paper from even reaching peer review and wasting a busy individuals time.

0cf8612b2e1e 3y ago
That was one particular dataset being requested. There are near infinite ways to characterize a material, and even the best lab is only going to have the time, resources, and expertise to collect and present some of them. I am not in the field, so I cannot comment on the specific relevance of the request, but in my area, there are absolutely some mandatory data inclusions which should block a paper from being published. Presumably this was not on that list.
- strangattractor 3y ago
  There are numerous ways to embargo data and make it available to reviewers before publication and to the public after publication. [1] This isn't a new idea. They simply did not do it. Nature is a "premier journal" known for publishing cutting edge research. Any data - in this case a bunch of numbers - used to support the conclusion should be made available. I am sure there are disciplines where the evidence would be difficult to provide directly - fossils, physical items etc. Maybe this PI did the experiment himself - thought he won a Noble - only to find he could not in fact reproduce the results after publication - oops. IMO Nature did not do its job. Just pointing that out. [1] https://plos.org/resource/how-to-store-and-manage-your-data/ https://plos.org/resource/how-to-store-and-manage-your-data/
BeetleB 3y ago
> Not having data is quite a significant gap. Requiring (raw) data is relatively new, and is only in a few fields. For most of the history of peer review data was not required. In fact, none of the PIs I worked in during grad school would ever agree to releasing the raw data. I'm pretty sure they're still not in most of physics. IMO, it's reasonable to require it, but understand that this is a relatively new requirement and not part of peer review culture.
- worik 3y ago
  > Requiring (raw) data is relatively new, and is only in a few fields. That on its own is scandalous I was reading of the "reproducibility crisis", in computing, in the nineties Science still wants to be the warm friendly club, whilst playing competitively for high personal stakes. Cannot be both. Fraud, deliberately cynical fraud and hopeful naive fraud, has been rampant for decades The journals benefit directly and seem to be utterly complicit. The system is dreadfully broken Nobody should wonder why people do not trust science
  - imchillyb 3y ago
    Science is a method. Science isn't a religion, nor is it a paper, thesis, or journal. It's a method for understanding the world around us through rigorous application of testing and retesting. Science can be trusted. Measurements and the testing and retesting can be trusted. It's people that can't be trusted. We call people in the field of Science, Scientists. Perhaps we should stop labeling fallible and untrustworthy humans as such. Science is trustworthy. Humans are not.
    - ants_everywhere 3y ago
      > Science can be trusted. Measurements and the testing and retesting can be trusted. It's people that can't be trusted. This is an important point and I think people should make it more often. A human scientist who is committing fraud is misrepresenting facts about the world. And the way you would accurately describe those facts is with science. It's not that science is in the wrong here, it's that the human is lying about what the science says. You want to be able to say "objectively, this guy lied about his research." But if you want to say that, then you need to be able to accurately describe both the world and the research so you can compute the diff between them. So if you throw out the science with the human scientist, then you've thrown out any objective means by which you were going to say the human scientist did anything wrong in the first place.
    - whelp_24 3y ago
      I strongly agree that there is a difference between science and "science". >Science can be trusted. Measurements and the testing and retesting can be trusted. I still wouldn't go so far. Science doesn't provide a pure understanding of the world it produces at best a model based on observations and axioms. These both can be wrong without politics,money, or personal bias. Our senses are fallible and sometimes you measured the wrong thing with the wrong tool. You shouldn't trust science, you scrutinize it and use what you find works until it doesn't (and then publish that).

0cf8612b2e1e 3y ago

That was one particular dataset being requested. There are near infinite ways to characterize a material, and even the best lab is only going to have the time, resources, and expertise to collect and present some of them. I am not in the field, so I cannot comment on the specific relevance of the request, but in my area, there are absolutely some mandatory data inclusions which should block a paper from being published. Presumably this was not on that list.

strangattractor 3y ago
There are numerous ways to embargo data and make it available to reviewers before publication and to the public after publication. [1] This isn't a new idea. They simply did not do it. Nature is a "premier journal" known for publishing cutting edge research. Any data - in this case a bunch of numbers - used to support the conclusion should be made available. I am sure there are disciplines where the evidence would be difficult to provide directly - fossils, physical items etc. Maybe this PI did the experiment himself - thought he won a Noble - only to find he could not in fact reproduce the results after publication - oops. IMO Nature did not do its job. Just pointing that out. [1] https://plos.org/resource/how-to-store-and-manage-your-data/ https://plos.org/resource/how-to-store-and-manage-your-data/

BeetleB 3y ago

> Not having data is quite a significant gap. Requiring (raw) data is relatively new, and is only in a few fields. For most of the history of peer review data was not required. In fact, none of the PIs I worked in during grad school would ever agree to releasing the raw data. I'm pretty sure they're still not in most of physics. IMO, it's reasonable to require it, but understand that this is a relatively new requirement and not part of peer review culture.

worik 3y ago
> Requiring (raw) data is relatively new, and is only in a few fields. That on its own is scandalous I was reading of the "reproducibility crisis", in computing, in the nineties Science still wants to be the warm friendly club, whilst playing competitively for high personal stakes. Cannot be both. Fraud, deliberately cynical fraud and hopeful naive fraud, has been rampant for decades The journals benefit directly and seem to be utterly complicit. The system is dreadfully broken Nobody should wonder why people do not trust science
- imchillyb 3y ago
  Science is a method. Science isn't a religion, nor is it a paper, thesis, or journal. It's a method for understanding the world around us through rigorous application of testing and retesting. Science can be trusted. Measurements and the testing and retesting can be trusted. It's people that can't be trusted. We call people in the field of Science, Scientists. Perhaps we should stop labeling fallible and untrustworthy humans as such. Science is trustworthy. Humans are not.
  - ants_everywhere 3y ago
    > Science can be trusted. Measurements and the testing and retesting can be trusted. It's people that can't be trusted. This is an important point and I think people should make it more often. A human scientist who is committing fraud is misrepresenting facts about the world. And the way you would accurately describe those facts is with science. It's not that science is in the wrong here, it's that the human is lying about what the science says. You want to be able to say "objectively, this guy lied about his research." But if you want to say that, then you need to be able to accurately describe both the world and the research so you can compute the diff between them. So if you throw out the science with the human scientist, then you've thrown out any objective means by which you were going to say the human scientist did anything wrong in the first place.
  - whelp_24 3y ago
    I strongly agree that there is a difference between science and "science". >Science can be trusted. Measurements and the testing and retesting can be trusted. I still wouldn't go so far. Science doesn't provide a pure understanding of the world it produces at best a model based on observations and axioms. These both can be wrong without politics,money, or personal bias. Our senses are fallible and sometimes you measured the wrong thing with the wrong tool. You shouldn't trust science, you scrutinize it and use what you find works until it doesn't (and then publish that).