3 ms·

That Chemist tried to use a paper published in Nature to synthesize something for his lab, but he found a number of potential concerning details in the paper an

That Chemist tried to use a paper published in Nature to synthesize something for his lab, but he found a number of potential concerning details in the paper and was unable to reproduce it using the published method. The authors and publication ghosted him and the paper was still up after 2 months. https://www.youtube.com/watch?v=-WPBtFTLZnM https://www.youtube.com/watch?v=-WPBtFTLZnM How do projects like this deal with papers published based on falsified data? Do they reproduce any of the source data themselves?

mkhorton 4y ago
> How do projects like this deal with papers published based on falsified data? Do they reproduce any of the source data themselves? I can't speak to this specific instance, but Materials Project does try to pay close attention to questions of reproducibility and provenance. Materials Project runs open-source repos[0] so that its methods can be verified, individual calculations are available via an API[1] and we also partner with NOMAD[2] to make larger files and calculation artifacts available for direct download. This is in addition to documenting methods via peer-reviewed papers, online docs, etc. This is not to say that issues of reproducibility don't still exist, or that we ourselves couldn't be doing better. It's a big problem in the community. [0] https://github.com/materialsproject https://github.com/materialsproject [1] https://api.materialsproject.org/docs https://api.materialsproject.org/docs [2] https://www.nomad-coe.eu https://www.nomad-coe.eu
dekhn 4y ago
Retraction watch. Btw many people (even trained) fail to reproduce legitimate work. It may not actually be fake or falsified, typically not being able to reproduce an experiment is insufficient evidence for retraction. I didn't watch the YouTube video.
- nl 4y ago
  Can confirm. People outside academia seem to believe that reproducibility is an explicit goal when publishing papers. Perhaps it should be, but generally speaking it isn't. Instead a paper is a record of the things the authors found interesting and novel when doing the work.
  - throwoutway 4y ago
    And if they’re objectively wrong, then what?
    - nl 4y ago
      "Wrong" is different to "fraudulent" If they are wrong then you publish a paper with conflicting results. If there is fraud (which is different to "I can't replicate the results") then you deal with the publisher of the paper.
      - infogulch 4y ago
        Well there's two problems, first is journals refusing to publish anything that conflicts with previous publications (a despicable practice that goes against the most fundamental principles of science that include curiosity and the willingness to be wrong and change, see also [1]), second is a reasonable expectation to get a response from the original authors when asked about a failure to replicate (even in the mode of "you're stupid and did it wrong"). [1]: The importance of stupidity in scientific research | https://news.ycombinator.com/item?id=31977918 https://news.ycombinator.com/item?id=31977918

mkhorton 4y ago

> How do projects like this deal with papers published based on falsified data? Do they reproduce any of the source data themselves? I can't speak to this specific instance, but Materials Project does try to pay close attention to questions of reproducibility and provenance. Materials Project runs open-source repos[0] so that its methods can be verified, individual calculations are available via an API[1] and we also partner with NOMAD[2] to make larger files and calculation artifacts available for direct download. This is in addition to documenting methods via peer-reviewed papers, online docs, etc. This is not to say that issues of reproducibility don't still exist, or that we ourselves couldn't be doing better. It's a big problem in the community. [0] https://github.com/materialsproject https://github.com/materialsproject [1] https://api.materialsproject.org/docs https://api.materialsproject.org/docs [2] https://www.nomad-coe.eu https://www.nomad-coe.eu

dekhn 4y ago

Retraction watch. Btw many people (even trained) fail to reproduce legitimate work. It may not actually be fake or falsified, typically not being able to reproduce an experiment is insufficient evidence for retraction. I didn't watch the YouTube video.

nl 4y ago
Can confirm. People outside academia seem to believe that reproducibility is an explicit goal when publishing papers. Perhaps it should be, but generally speaking it isn't. Instead a paper is a record of the things the authors found interesting and novel when doing the work.
- throwoutway 4y ago
  And if they’re objectively wrong, then what?
  - nl 4y ago
    "Wrong" is different to "fraudulent" If they are wrong then you publish a paper with conflicting results. If there is fraud (which is different to "I can't replicate the results") then you deal with the publisher of the paper.
    - infogulch 4y ago
      Well there's two problems, first is journals refusing to publish anything that conflicts with previous publications (a despicable practice that goes against the most fundamental principles of science that include curiosity and the willingness to be wrong and change, see also [1]), second is a reasonable expectation to get a response from the original authors when asked about a failure to replicate (even in the mode of "you're stupid and did it wrong"). [1]: The importance of stupidity in scientific research | https://news.ycombinator.com/item?id=31977918 https://news.ycombinator.com/item?id=31977918