7 ms·

Retraction of Roomtemperature superconductivity in a carbonaceous sulfur hydride

72 points by YakBizzarro 4y ago

zalo 4y ago
That’s a shame; I hope they’re allowed to come back in a revised article. …or perhaps Nature felt it was too early for a discovery of this magnitude to be published ;-)
- Jolter 4y ago
  I’m sure the experiment could be repeated with correct data processing. The authors probably won’t do it, unless they think it will provide a positive result.
  - pbhjpbhj 4y ago
    That sort of thinking is surely a great loss to scientific progress? You need a few goes by different labs to show failure, so you know that if you were going to try that technique you need to think again, surely? Sometimes one should probably even do experiments that seem self-evidently futile.
nyokodo 4y ago
This is why it’s best to adopt a skeptical attitude about published science at least until after a reasonable time period has passed for general scrutiny and retraction but maybe always. This is especially important if the subject is exciting, surprising, relates to some political controversy, or anything else that tends to provoke motivated reasoning.
- dbingham 4y ago
  Honestly, we too often forget that science works in the aggregate. There are far, far too many things that can go wrong in any individual study - from biases, to mistakes, to outright fraud, to just bad luck and weird statistical flukes. It's only after we've had several replications that we can really say we think something might be true, and only after many replications that we can say we "know" something. This aspect of science seems to have been lost both in the public's imagination and - all too often - in the institutions own understanding and approach.
jinto36 4y ago
I'm not sure why then they couldn't just apply a more standard background subtraction method and show that the result still holds. Note that this approach was "room temperature" but under very (very) high pressure.
- beanjuice 4y ago
  In retracting the work, the editors at Nature were in some way certain "why" they couldn't apply a more standard method.
- jjk166 4y ago
  It's not that they did the background subtraction in a non-standard way, it's that they didn't have a measurement of the background to subtract (it being too hard to measure directly), so they instead estimated what the background would look like and subtracted that. Some of their peers are claiming their estimate is bad, but the researchers are sticking by it.
beanjuice 4y ago
In passing, it is just a retracted paper. However, the lab website of the corresponding author offers a comprehensive time machine to the impact the announcement of this work originally had [1]. It seems to be that, in a collaboration between three graduate students, some background data which was foundational to the interpretation of other findings may have been fabricated. As a researcher, all i can think when a major retraction happens is: what a nightmare. Every author of a manuscript should have full faith and knowledge of the results and presented data, but in a massive piece of work such as this, how can you? Imagine for a second, one graduate student doesn't want to face the wrath of an Assistant Professor one day, on purpose or not mishandles data. A year or two later it turns out you have not only wasted your time, but also the time of 500+ citing paper authors, numerous grant proposals, etc. Alongside the Alzheimer's retraction this year, the thirst for impact factor and the problems this causes seems to be yet another unsolvable problem in academia. [1] https://labsites.rochester.edu/dias/ https://labsites.rochester.edu/dias/
- dbingham 4y ago
  Not unsolvable - just difficult. Systematic problems driven by social incentives are very difficult to solve. But they can be solved. Especially if the institutions in question get behind solving them. In the case of science, a lot of the bad incentives are created by a combination of the institutions (university administrations and granting agencies) and by the scientists themselves. It'll take both working together to change the incentives and solve the problems. But it can be done.
  - uup 4y ago
    Fabricating data is fraud. The reputation of the university is harmed. The public is harmed by potentially providing grants for follow up research. Other researchers are harmed by wasting their time. It seems like one potential fix would be to prosecute people who perpetrate such fraud.
    - pca006132 4y ago
      Who should you prosecute in this case? The PI? The students?
routeroff 4y ago
That was a bit expected, and the reason given by Nature are not the whole story. Some of the same authors also claimed to have found metallic hydrogen [0], which is also highly controversial since the proof hold mainly on pictures taken using a Iphone camera. [0] https://www.science.org/doi/full/10.1126/science.aal1579 https://www.science.org/doi/full/10.1126/science.aal1579
routeroff 4y ago
This short article in Science [1] give more information and summarize the whole story. [1] https://www.science.org/content/article/something-seriously-wrong-room-temperature-superconductivity-study-retracted https://www.science.org/content/article/something-seriously-...
Gatsky 4y ago
Not sure how Nature lets through a paper like this without mandating the raw data is included and a detailed protocol to reproduce the results.
- tom-thistime 4y ago
  I don't think I've ever seen a paper that included much raw data. It's possible this may be a good suggestion, but it's not the way things are currently done.
  - Gatsky 4y ago
    Maybe not typically, but room temp superconductivity is a nobel prize level claim, and it seems that the experiment is not fully specified without the raw data.
    - p_j_w 4y ago
      The scientific community generally does a pretty good job of shooting down big claims like this that get published (see: Pons and Fleischmann), so what's the huge benefit? On top of this, if it's to have any value, the reviewers of the article would need to recreate whatever code was used to process the raw data OR audit the existing code to make sure it made sense, both of which are pretty heavy duty tasks. Reviewers don't get paid, so this is a good way to ensure no one wants to review articles for your journal.
      - jimmydorry 4y ago
        If there is substantial interest in the paper after it has been published, it would be a lot easier to verify with the raw data than without. Nothing about the peer review process necessarily needs change beyond immediately rejecting all papers that do not include the minimum amount of data to validate their claims (even if said data is simply stored and not immediately reviewed).
  - rsfern 4y ago
    Many journals (including a lot of the nature ones) are requiring data and code on publication now Part of the problem is that as far as I can tell it’s on the reviewers to flag noncompliance, and a lot of times groups won’t actually publish code and datasets before the paper is accepted and assigned a DOI. So really it’s a culture change around the whole publishing process that’s needed, IMO
tom-thistime 4y ago
If you haven't worked in science, please keep in mind that the literature has never been seen as holy writ. Wrong stuff gets published all the time. You have to read critically. Not excusing sloppy work or whatever it was. Just pointing out to any laypeople out there that science has never had (and never required) a perfect professional literature. Far from it.
- dbingham 4y ago
  Not just that, but it would be impossible to achieve. One study should never be taken as truth - but rather a single data point among many. It's only when we have many data points that give us a trend line that we can begin to say we know something.
  - TillE 4y ago
    Right, any casual reader who wants good information should look for meta analyses. You should almost never draw conclusions from a single study.
- Pulcinella 4y ago
  My copy of the CRC Handbook of Chemistry and Physics (basically an encyclopedia of scientific facts. Things like melting points, conductivity, etc.) has an opening from the current editor explaining that when he took over as editor he tried to track down the sources of a lot of the data. Unfortunately a lot of the sources for the data wasn’t listed, missing, didn’t seem to exist, or existed but was incorrect. So he undertook a program of re-doing a lot of the studies and finding new, accurate sources. Definitely gave me sort of cyberpunk vibes of “this data just showed up one day and nobody knows where it came from.” Edit: I’ve also run across cases where data is just incorrect and you don’t know how. E.g. a chemical ordered form a supplier is listed as having a solubility of X you the most you can ever get to dissolve is Y and X is something like 13.7 times higher than Y so it’s not like a simple misplaced decimal.
- bobsmooth 4y ago
chrisbrandow 4y ago
Due to the typically discrete nature of the phenomnenon in question, chemistry and molecular-scale physics seems to the worst aspects of the reproducibility crisis in science more broadly. It’s just harder to hide either mistakes or deceptions. The experiments can typically be setup by another single lab hoping to build on the work, rather than requiring the collaboration of countless others. It’s not a terrible outcome that a paper was published then retracted merely 2 years later as opposed to 1-2 decades later.
deleted 4y ago
[deleted]
moomin 4y ago
So they've set up a whole new company on the basis of refuted experimental result. I imagine it will have the ghost of Martin Fleischmann on the board.
- bigbillheck 4y ago
  > So they've set up a whole new company on the basis of refuted experimental result Nothing new about that.
- jjk166 4y ago
  The experimental result has not been refuted, the whole issue is that some scientists are claiming there isn't enough information to attempt to refute it.
  - DFHippie 4y ago
    One research quoted said he had tried and failed 6 times to reproduce the results.
    - jjk166 4y ago
      Because the formula was not published.
shishy 4y ago
Retractions are a healthy part of the process -- they shouldn't be frowned upon to the extent that they are. Me and my team just submitted a paper outlining a system we're using at scite to detect retractions more quickly from the titles themselves (instead of manual curation, or relying on publisher updates, which can be spotty / inconsistent). Happy to share details if anyone is interested, since that data will be open for free. email: ashish AT scite.ai
hartjer 4y ago
Reminds of Jan Hendrik Shrön https://www.youtube.com/watch?v=nfDoml-Db64 https://www.youtube.com/watch?v=nfDoml-Db64 https://en.wikipedia.org/wiki/Sch%C3%B6n_scandal https://en.wikipedia.org/wiki/Sch%C3%B6n_scandal
kalimanzaro 4y ago
Yesterday (before this came up on HN) I came across in my daily paper chase a follow up paper from the lead author Ranga. Looks like the same fossil superconductor, but they manage to add another sensational detail: the superconductor is composed of dissociated molecules(?!!!). Truly shades of another Hendrik Schon[0]. https://pubs.rsc.org/en/content/articlelanding/2022/cc/d2cc03170a https://pubs.rsc.org/en/content/articlelanding/2022/cc/d2cc0... So far, it has not been retracted. One wonders whether anybody at the journals ever glances at the maxim "If it's too good to be true, have someone check the data." [0]https://en.m.wikipedia.org/wiki/Sch%C3%B6n_scandal https://en.m.wikipedia.org/wiki/Sch%C3%B6n_scandal Footnote added: the toogoodtobetrue bit in the original paper was that the transitions were a bit too sharp for a real superconductor. That was the main thing that got this started.

72 points by YakBizzarro 4y ago

zalo 4y ago

That’s a shame; I hope they’re allowed to come back in a revised article. …or perhaps Nature felt it was too early for a discovery of this magnitude to be published ;-)

Jolter 4y ago
I’m sure the experiment could be repeated with correct data processing. The authors probably won’t do it, unless they think it will provide a positive result.
- pbhjpbhj 4y ago
  That sort of thinking is surely a great loss to scientific progress? You need a few goes by different labs to show failure, so you know that if you were going to try that technique you need to think again, surely? Sometimes one should probably even do experiments that seem self-evidently futile.

nyokodo 4y ago

This is why it’s best to adopt a skeptical attitude about published science at least until after a reasonable time period has passed for general scrutiny and retraction but maybe always. This is especially important if the subject is exciting, surprising, relates to some political controversy, or anything else that tends to provoke motivated reasoning.

dbingham 4y ago
Honestly, we too often forget that science works in the aggregate. There are far, far too many things that can go wrong in any individual study - from biases, to mistakes, to outright fraud, to just bad luck and weird statistical flukes. It's only after we've had several replications that we can really say we think something might be true, and only after many replications that we can say we "know" something. This aspect of science seems to have been lost both in the public's imagination and - all too often - in the institutions own understanding and approach.

jinto36 4y ago

I'm not sure why then they couldn't just apply a more standard background subtraction method and show that the result still holds. Note that this approach was "room temperature" but under very (very) high pressure.

beanjuice 4y ago
In retracting the work, the editors at Nature were in some way certain "why" they couldn't apply a more standard method.
jjk166 4y ago
It's not that they did the background subtraction in a non-standard way, it's that they didn't have a measurement of the background to subtract (it being too hard to measure directly), so they instead estimated what the background would look like and subtracted that. Some of their peers are claiming their estimate is bad, but the researchers are sticking by it.

beanjuice 4y ago

In passing, it is just a retracted paper. However, the lab website of the corresponding author offers a comprehensive time machine to the impact the announcement of this work originally had [1]. It seems to be that, in a collaboration between three graduate students, some background data which was foundational to the interpretation of other findings may have been fabricated. As a researcher, all i can think when a major retraction happens is: what a nightmare. Every author of a manuscript should have full faith and knowledge of the results and presented data, but in a massive piece of work such as this, how can you? Imagine for a second, one graduate student doesn't want to face the wrath of an Assistant Professor one day, on purpose or not mishandles data. A year or two later it turns out you have not only wasted your time, but also the time of 500+ citing paper authors, numerous grant proposals, etc. Alongside the Alzheimer's retraction this year, the thirst for impact factor and the problems this causes seems to be yet another unsolvable problem in academia. [1] https://labsites.rochester.edu/dias/ https://labsites.rochester.edu/dias/

dbingham 4y ago
Not unsolvable - just difficult. Systematic problems driven by social incentives are very difficult to solve. But they can be solved. Especially if the institutions in question get behind solving them. In the case of science, a lot of the bad incentives are created by a combination of the institutions (university administrations and granting agencies) and by the scientists themselves. It'll take both working together to change the incentives and solve the problems. But it can be done.
- uup 4y ago
  Fabricating data is fraud. The reputation of the university is harmed. The public is harmed by potentially providing grants for follow up research. Other researchers are harmed by wasting their time. It seems like one potential fix would be to prosecute people who perpetrate such fraud.
  - pca006132 4y ago
    Who should you prosecute in this case? The PI? The students?

routeroff 4y ago

That was a bit expected, and the reason given by Nature are not the whole story. Some of the same authors also claimed to have found metallic hydrogen [0], which is also highly controversial since the proof hold mainly on pictures taken using a Iphone camera. [0] https://www.science.org/doi/full/10.1126/science.aal1579 https://www.science.org/doi/full/10.1126/science.aal1579

routeroff 4y ago

This short article in Science [1] give more information and summarize the whole story. [1] https://www.science.org/content/article/something-seriously-wrong-room-temperature-superconductivity-study-retracted https://www.science.org/content/article/something-seriously-...

Gatsky 4y ago

Not sure how Nature lets through a paper like this without mandating the raw data is included and a detailed protocol to reproduce the results.

tom-thistime 4y ago
I don't think I've ever seen a paper that included much raw data. It's possible this may be a good suggestion, but it's not the way things are currently done.
- Gatsky 4y ago
  Maybe not typically, but room temp superconductivity is a nobel prize level claim, and it seems that the experiment is not fully specified without the raw data.
  - p_j_w 4y ago
    The scientific community generally does a pretty good job of shooting down big claims like this that get published (see: Pons and Fleischmann), so what's the huge benefit? On top of this, if it's to have any value, the reviewers of the article would need to recreate whatever code was used to process the raw data OR audit the existing code to make sure it made sense, both of which are pretty heavy duty tasks. Reviewers don't get paid, so this is a good way to ensure no one wants to review articles for your journal.
    - jimmydorry 4y ago
      If there is substantial interest in the paper after it has been published, it would be a lot easier to verify with the raw data than without. Nothing about the peer review process necessarily needs change beyond immediately rejecting all papers that do not include the minimum amount of data to validate their claims (even if said data is simply stored and not immediately reviewed).
- rsfern 4y ago
  Many journals (including a lot of the nature ones) are requiring data and code on publication now Part of the problem is that as far as I can tell it’s on the reviewers to flag noncompliance, and a lot of times groups won’t actually publish code and datasets before the paper is accepted and assigned a DOI. So really it’s a culture change around the whole publishing process that’s needed, IMO

tom-thistime 4y ago

If you haven't worked in science, please keep in mind that the literature has never been seen as holy writ. Wrong stuff gets published all the time. You have to read critically. Not excusing sloppy work or whatever it was. Just pointing out to any laypeople out there that science has never had (and never required) a perfect professional literature. Far from it.

dbingham 4y ago
Not just that, but it would be impossible to achieve. One study should never be taken as truth - but rather a single data point among many. It's only when we have many data points that give us a trend line that we can begin to say we know something.
- TillE 4y ago
  Right, any casual reader who wants good information should look for meta analyses. You should almost never draw conclusions from a single study.
Pulcinella 4y ago
My copy of the CRC Handbook of Chemistry and Physics (basically an encyclopedia of scientific facts. Things like melting points, conductivity, etc.) has an opening from the current editor explaining that when he took over as editor he tried to track down the sources of a lot of the data. Unfortunately a lot of the sources for the data wasn’t listed, missing, didn’t seem to exist, or existed but was incorrect. So he undertook a program of re-doing a lot of the studies and finding new, accurate sources. Definitely gave me sort of cyberpunk vibes of “this data just showed up one day and nobody knows where it came from.” Edit: I’ve also run across cases where data is just incorrect and you don’t know how. E.g. a chemical ordered form a supplier is listed as having a solubility of X you the most you can ever get to dissolve is Y and X is something like 13.7 times higher than Y so it’s not like a simple misplaced decimal.
bobsmooth 4y ago

chrisbrandow 4y ago

Due to the typically discrete nature of the phenomnenon in question, chemistry and molecular-scale physics seems to the worst aspects of the reproducibility crisis in science more broadly. It’s just harder to hide either mistakes or deceptions. The experiments can typically be setup by another single lab hoping to build on the work, rather than requiring the collaboration of countless others. It’s not a terrible outcome that a paper was published then retracted merely 2 years later as opposed to 1-2 decades later.

deleted 4y ago

[deleted]

moomin 4y ago

So they've set up a whole new company on the basis of refuted experimental result. I imagine it will have the ghost of Martin Fleischmann on the board.

bigbillheck 4y ago
> So they've set up a whole new company on the basis of refuted experimental result Nothing new about that.
jjk166 4y ago
The experimental result has not been refuted, the whole issue is that some scientists are claiming there isn't enough information to attempt to refute it.
- DFHippie 4y ago
  One research quoted said he had tried and failed 6 times to reproduce the results.
  - jjk166 4y ago
    Because the formula was not published.

shishy 4y ago

Retractions are a healthy part of the process -- they shouldn't be frowned upon to the extent that they are. Me and my team just submitted a paper outlining a system we're using at scite to detect retractions more quickly from the titles themselves (instead of manual curation, or relying on publisher updates, which can be spotty / inconsistent). Happy to share details if anyone is interested, since that data will be open for free. email: ashish AT scite.ai

hartjer 4y ago

Reminds of Jan Hendrik Shrön https://www.youtube.com/watch?v=nfDoml-Db64 https://www.youtube.com/watch?v=nfDoml-Db64 https://en.wikipedia.org/wiki/Sch%C3%B6n_scandal https://en.wikipedia.org/wiki/Sch%C3%B6n_scandal

kalimanzaro 4y ago

Yesterday (before this came up on HN) I came across in my daily paper chase a follow up paper from the lead author Ranga. Looks like the same fossil superconductor, but they manage to add another sensational detail: the superconductor is composed of dissociated molecules(?!!!). Truly shades of another Hendrik Schon[0]. https://pubs.rsc.org/en/content/articlelanding/2022/cc/d2cc03170a https://pubs.rsc.org/en/content/articlelanding/2022/cc/d2cc0... So far, it has not been retracted. One wonders whether anybody at the journals ever glances at the maxim "If it's too good to be true, have someone check the data." [0]https://en.m.wikipedia.org/wiki/Sch%C3%B6n_scandal https://en.m.wikipedia.org/wiki/Sch%C3%B6n_scandal Footnote added: the toogoodtobetrue bit in the original paper was that the transitions were a bit too sharp for a real superconductor. That was the main thing that got this started.