16 ms·

Everyone in this thread needs to understand that particularly for materials science, and particularly when there is bad methodology documentation, it is really,

Everyone in this thread needs to understand that particularly for materials science, and particularly when there is bad methodology documentation, it is really, really normal for early replication attempts to fail. What works in one lab will often need to be adjusted to work in another, with different equipment, altitude, humidity, all sorts of stuff. Making it even worse, apparently the original team can't even get production correct in more than 10% of runs! Combined with the slipshod methodology, doesn't this make you feel like it was silly to release the arXiv paper when they did? Well, the LK-99 team agree, according to them it was a rogue industrial scientist that they'd apparently fired four months ago who published it to arXiv with himself listed as third author. They wanted more time to nail down sample production and otherwise make a paper to their standards before release, but it effectively leaked and to avoid having credit taken from them for potentially months (or like, forever - science can be brutal) some of them decided to publish what they had on hand within a few hours. This sucks for them, a lot. I don't believe that their intention was to put out work with errors in it or incomplete methodology. Secondly, this all happened two days ago! If the methodology was perfect, I still wouldn't expect good replication results back within two (part non-business!) days. I don't know if this material is legit. I really hope it is. But the process of figuring that out could potentially take months (or more), and a two-day-later failed to replicate is not a death sentence. As much as it sucks, this is really just going to take time to prove it's not a superconductor. If it is a superconductor, we may not know for a while either, unless one of the influencers/"makers" attempting to reproduce the material succeeds and posts some good convincing video of flux pinning or other Meissner effect stuff (that person is going to go insanely viral, if it happens).

dmvdoug 3y ago
Not just materials science though. I’m a longtime reader of Derek Lowe’s In the Pipeline blog, and he often emphasizes the trickiness of lab work and lab techniques. But the model we have of science from elementary and high school doesn’t give us room for the random, ephemeral things that come into play in actual experimental work. Our model is methods equal result, every time, even though jiggling your foot a little bit to get the feeling back may have somehow made your small molecule production have less binding affinity or something. And yes, eventually you figure that out and stop jiggling your foot but two years have passed.
- lucubratory 3y ago
  Absolutely, ultimate failures to replicate are generally multiple labs across dozens and dozens of attempts and continuous communication with the authors. And yeah, it's a thing in science in general, materials science is just particularly susceptible because of the nature of the work.
- tetha 3y ago
  NileRed and other Maker Channels are great to demonstrate that. In every video, there's 1 - 2 of these situations - Nile has been following some documented, established procedure which should result in a specific reaction and it looks decent to the layman - and then it just doesn't work at all. And then he re-does pretty much the same thing and it works. I can't imagine how frustrating that can be. And then you have to do it with something you don't even know works. Kind of like dealing with obscure retro computers. Does it still work? No one knows. What does it do if it works? No one knows. Can it even function or are parts missing? That's right - no one knows.
  - teaearlgraycold 3y ago
    Link to NileRed video: https://www.youtube.com/watch?v=RS7gyZJg5nc https://www.youtube.com/watch?v=RS7gyZJg5nc
  - MichaelZuo 3y ago
    Well to be fair it's very difficult to nail down every possible parameter to 99.99% accuracy and precision from the original successful experiment. There's a lot of randomness even in a highly controlled, well funded, modern laboratory setting, let alone less well equipped facilities.
  - whimsicalism 3y ago
    It is incredibly frustrating, my experience in an experimental physics lab involved in creation of certain types of wafers for SEM characterization, there are so many things that can go wrong for certain materials. just a certain speck of dust being in the wrong place at the wrong time It is why I quit experimental physics.
deleted 3y ago
[deleted]
vasco 3y ago
If research moves more to arxiv + discussion forums rather than paid journals, sharing research results earlier seems like a good thing, not bad. If they didn't publish early, when they published maybe things would be clearer for other researchers. But by publishing earlier other people have the choice to engage with it or wait for later. Seems like many people decided to not wait, so it seems like it was a good choice?
- was_a_dev 3y ago
  I think this would vary greatly on the field. This example of results is one of the better examples imo
- brutusborn 3y ago
  I think this is the best direction for science generally: publish everything open and early. It might be naive, but I would love an environment where everyone self published or arxiv papers/reviews/datasets/models. Then journals could publish the best papers/review preprints. I imagine a world where journals beg researchers for the rights to publish their paper after seeing an excellent preprint. Or a journal finds an interesting comment on PubPeer and pays for the author to expand it into a more detailed review.
  - WanderPanda 3y ago
    Nice counterfactual! Do you think it is just a path dependency that we didn‘t end up in the (local) minimum that you mention?
    - brutusborn 3y ago
      Yes I think we will end there someday but are still overcoming a publishing model historically based on physical journals. Open science becoming more common shows that progress is occurring. It will take a while before academics and systems upgrade skills and infrastructure. When they do, I am excited for enhanced capabilities like continuous science. Imagine a website with a study result with P values which continuously update as data is published and integrated into the existing analysis! https://towardsdatascience.com/continuous-science-6b5ed833ab23 https://towardsdatascience.com/continuous-science-6b5ed833ab...
deleted 3y ago
[deleted]
joefigura 3y ago
This comment intrigued me, so I pulled together some information. The first paper submitted is titled "The First Room-Temperature Ambient-Pressure Superconductor." It lists three authors: Sukbae Lee, Ji-Hoon Kim, and Young-Wan Kwon. Its timestamp is Saturday, July 22, 2023 at 07:51:19 UTC. [1] The second paper submitted is titled "Superconductor Pb10−xCux(PO4)6O showing levitation at room temperature and atmospheric pressure and mechanism." This paper lists six authors: Sukbae Lee, Jihoon Kim, Hyun-Tak Kim, Sungyeon Im, SooMin An, Keun Ho Auh [2]. Its timestamp is Saturday, July 22nd, 2023 at 10:11:28 UTC, or two hours and twenty minutes after the first paper. In both papers the first author is Sukbae Lee and the second author is Jihoon Kim, and in both their affiliation is given as "Quantum Energy Research center, Inc." in Seoul. The first paper posted has Young-Wan Kwon as third author. The second paper does not have Young-Wan Kwon as an author, and has four additional authors with various affiliations. The second paper appears to have been was prepared in LaTeX, and the first paper appears to have been prepared in Word. The title and abstract of the first paper explicitly claim the creation of the world's first room temperature and pressure superconductor. The title and abstract of the second paper don't explicitly claim demonstration of the first superconductor, though they use some terminology that sounds like superconducting properties. The accusation is that Young-Wan Kwon published the first paper without the consent of the rest of the LK-99 team, listed himself as third author, and left off the other four. Two hours later, the rest of the LK-99 team stuffed as much as they had into the second paper, and released it as soon as possible. To me that totally seems like what happened. It explains why there are two different papers from the same group submitted on the same subject on the same day, and it explains why the author lists are different between the two. I haven't yet looked in detail, but I'm betting it also explains a lot of the oddities that people have noted in the first two papers. This also makes me way more excited about the possibility these claims are legit. The information so far is consistent with a research group that was forced to publish early, and who produced a superconductor through a fabrication process that is a bit tricky. There's nowhere near enough evidence to conclude LK-99 is a room temperature superconductor. But one failed replication doesn't prove LK-99 isn't a superconductor - if the fabrication process is finicky we'd expect to see a few dozen failed reproductions and a few successful reproductions [1] https://arxiv.org/abs/2307.12008 https://arxiv.org/abs/2307.12008 [2] https://arxiv.org/abs/2307.12037 https://arxiv.org/abs/2307.12037
deleted 3y ago
[deleted]
bruce511 3y ago
It's not dissimilar to the way food recipe books work. Creating a dish, replicating the dish, writing a recipe, getting competent cooks to follow the recipe, getting non-competant cooks to follow it, and so on. [1] It all leads to multiple revisions and takes a lot of time. And this is for -food-. Imagine doing it for something a lot less tolerant of variation. [1] old recipes start with "pluck 2 chickens" because, you know, someone followed the recipe and didn't pluck the chickens...
comboy 3y ago
This is a good summary: https://youtu.be/RjzL9cS3VW8?t=32 https://youtu.be/RjzL9cS3VW8?t=32 TLDW it seems very unlikely that they have what they claim
- lucubratory 3y ago
  In Sabine's opinion. I've seen Sabine's video when it was posted and intend no disrespect to her. The predominant theoretical explanation for the alleged new superconductor is that it may be a new type of superconductor, not type I or type II, or it may be presenting in unusual ways. There have been a bunch of confusing theoretical prodictions, like anisotropic conduction (how would that affect the way levitation looks)?, chance-based copper substitution in the lattice that could lead to superconducting clusters in what is otherwise just normal lead apatite (how would that affect the way levitation looks in a bulk chunk like that?), etc. Plenty of people who know the theory of superconductors are sceptical. They're not wrong about the theory. It's also worth pointing out that every previous time a new superconductor critical temp limit was broken in a major way, superconductor theory saying it shouldn't be possible was a cause for early scepticism amongst those who knew the theory. Sabine's gut reactions in a YouTube video doesn't make the result very unlikely. It doesn't make her wrong either, she's right that all existing superconductors, if pure and homogeneous, would not behave like that on a magnet. It is also the case that all existing superconductors do not superconduct at room temperature and ambient pressure. It is also the case that (absent fraud, which is on everyone's mind after Ranga Dias) we don't know of any material that could plausible behave the way that video shows something behaving. The only thing I can think of that could behave that way would be impure/heterogeneous diamagnetic material, but like 15x as diamagnetic as the next highest non-superconductor, which means if that material exists we don't know about it. If it was fraud, it would be with a magnetised sample on a magnet like that. The reason I think fraud is the only option if it was a magnetised sample is because that's a really, really obvious explanation for the effect and they have to have measured the sample for magnetism before putting it on a magnet. They would also literally feel it/see it/hear it while placing the sample on the magnet. Also it would almost certainly have to be composed of different elements than they describe, because lead apatite with copper definitely should not be ferromagnetic at all.
michaelt 3y ago
> Combined with the slipshod methodology, doesn't this make you feel like it was silly to release the arXiv paper when they did? Well, the LK-99 team agree As I understand it, the 99 in the name is because they first discovered it in 1999. Seems to me that if you didn't manage to nail down the process in 24 years, it's probably time to publish and let someone else have a go.
- londons_explore 3y ago
  And if this technology is on the critical path to solving world hunger/curing cancer or ending wars, then those things have been delayed 24 years, and millions of people are dead, all so these guys could have a shot at a paper in Nature. Humans really are selfish sometimes.
  - simmerup 3y ago
    Assuming the worst of people is bad for your mental health
- guerrilla 3y ago
  It sounds like you don't know the history. They were doing other unrelated things in the meantime, not working in this.
  - michaelt 3y ago
    I'm aware of that. But if I was an early-career researcher looking to establish myself, secure funding and maybe a tenure-track position, and I had a nobel-prize-winning discovery up my sleeve, I can't imagine many things that would take precedence. I mean, obviously whatever happened happened. It just seems strange to simultaneously claim to have discovered something 24 years ago, and to have been rushed into publishing prematurely.
    - guerrilla 3y ago
      > It just seems strange to simultaneously claim to have discovered something 24 years ago, and to have been rushed into publishing prematurely. They had to get other jobs to survive. Somone else forced their hand with the publication. There's nothing incosistent about that.
DannyBee 3y ago
The thing is - there is a totally easy and simple mechanism to create early-stage believability. Take a video of you producing and testing it, with a mobile phone. It only works 10% of the time? Not an issue, post 10 videos. Whatever. Would it be perfect? No, of course it could be faked. But i guaranteed people would be a ton less skeptical and willing to push hard on this if the original authors had done this sort of thing. It's honestly not that hard to set up a tripod and a mobile phone, and even in the circumstances here, where they apparently got pushed to publish quickly, it could still show that it works. It also would help those trying to reproduce see exactly what you did, in detail.
- bluGill 3y ago
  Production takes more than a day - I don't know of any phone that can take videos lasting that long (if people are even willing to watch it). You can of course take several videos and splice them together. Worse yet, I can with a bit of trick photography and standard liquid nitrogen cooled super conductors make a video that looks like success even though I'm working with colored wax for materials in the production process. (liquid nitrogen is hard to disguise, but with some trick photography and multiple takes I can do it). Or I can skip all that and spend some time in blender to do the above via CGI. (well I don't know blender that well, and most blender animations are obvious, but someone with a few months could pull this off)
  - DannyBee 3y ago
    Ignore mobile phones, put in a nest, whatever. It is not that hard or expensive to record footage for a few days at a time having a continuous overwrite. You likely don't even need to go that far -Most universities have cameras everywhere, anyway, including plenty of universities that have cameras in every lab of this kind. It is also done by millions of home users all the time. As I said, yes, things can be faked, but who cares that’s not the point
- mhb 3y ago
  Are you suggesting that they take phone video of all their experiments over the course of multiple years? Or just the one they know will succeed?
throwawaymaths 3y ago
we used to joke in the chemistry lab that someone once tried to reproduce a reaction that was done in india. Procedure called for boiling ether, room temperature, WTF. Of course, room temperature in india can get up to 40C.

dmvdoug 3y ago

Not just materials science though. I’m a longtime reader of Derek Lowe’s In the Pipeline blog, and he often emphasizes the trickiness of lab work and lab techniques. But the model we have of science from elementary and high school doesn’t give us room for the random, ephemeral things that come into play in actual experimental work. Our model is methods equal result, every time, even though jiggling your foot a little bit to get the feeling back may have somehow made your small molecule production have less binding affinity or something. And yes, eventually you figure that out and stop jiggling your foot but two years have passed.

lucubratory 3y ago
Absolutely, ultimate failures to replicate are generally multiple labs across dozens and dozens of attempts and continuous communication with the authors. And yeah, it's a thing in science in general, materials science is just particularly susceptible because of the nature of the work.
tetha 3y ago
NileRed and other Maker Channels are great to demonstrate that. In every video, there's 1 - 2 of these situations - Nile has been following some documented, established procedure which should result in a specific reaction and it looks decent to the layman - and then it just doesn't work at all. And then he re-does pretty much the same thing and it works. I can't imagine how frustrating that can be. And then you have to do it with something you don't even know works. Kind of like dealing with obscure retro computers. Does it still work? No one knows. What does it do if it works? No one knows. Can it even function or are parts missing? That's right - no one knows.
- teaearlgraycold 3y ago
  Link to NileRed video: https://www.youtube.com/watch?v=RS7gyZJg5nc https://www.youtube.com/watch?v=RS7gyZJg5nc
- MichaelZuo 3y ago
  Well to be fair it's very difficult to nail down every possible parameter to 99.99% accuracy and precision from the original successful experiment. There's a lot of randomness even in a highly controlled, well funded, modern laboratory setting, let alone less well equipped facilities.
- whimsicalism 3y ago
  It is incredibly frustrating, my experience in an experimental physics lab involved in creation of certain types of wafers for SEM characterization, there are so many things that can go wrong for certain materials. just a certain speck of dust being in the wrong place at the wrong time It is why I quit experimental physics.

deleted 3y ago

[deleted]

vasco 3y ago

If research moves more to arxiv + discussion forums rather than paid journals, sharing research results earlier seems like a good thing, not bad. If they didn't publish early, when they published maybe things would be clearer for other researchers. But by publishing earlier other people have the choice to engage with it or wait for later. Seems like many people decided to not wait, so it seems like it was a good choice?

was_a_dev 3y ago
I think this would vary greatly on the field. This example of results is one of the better examples imo
brutusborn 3y ago
I think this is the best direction for science generally: publish everything open and early. It might be naive, but I would love an environment where everyone self published or arxiv papers/reviews/datasets/models. Then journals could publish the best papers/review preprints. I imagine a world where journals beg researchers for the rights to publish their paper after seeing an excellent preprint. Or a journal finds an interesting comment on PubPeer and pays for the author to expand it into a more detailed review.
- WanderPanda 3y ago
  Nice counterfactual! Do you think it is just a path dependency that we didn‘t end up in the (local) minimum that you mention?
  - brutusborn 3y ago
    Yes I think we will end there someday but are still overcoming a publishing model historically based on physical journals. Open science becoming more common shows that progress is occurring. It will take a while before academics and systems upgrade skills and infrastructure. When they do, I am excited for enhanced capabilities like continuous science. Imagine a website with a study result with P values which continuously update as data is published and integrated into the existing analysis! https://towardsdatascience.com/continuous-science-6b5ed833ab23 https://towardsdatascience.com/continuous-science-6b5ed833ab...

deleted 3y ago

[deleted]

joefigura 3y ago

This comment intrigued me, so I pulled together some information. The first paper submitted is titled "The First Room-Temperature Ambient-Pressure Superconductor." It lists three authors: Sukbae Lee, Ji-Hoon Kim, and Young-Wan Kwon. Its timestamp is Saturday, July 22, 2023 at 07:51:19 UTC. [1] The second paper submitted is titled "Superconductor Pb10−xCux(PO4)6O showing levitation at room temperature and atmospheric pressure and mechanism." This paper lists six authors: Sukbae Lee, Jihoon Kim, Hyun-Tak Kim, Sungyeon Im, SooMin An, Keun Ho Auh [2]. Its timestamp is Saturday, July 22nd, 2023 at 10:11:28 UTC, or two hours and twenty minutes after the first paper. In both papers the first author is Sukbae Lee and the second author is Jihoon Kim, and in both their affiliation is given as "Quantum Energy Research center, Inc." in Seoul. The first paper posted has Young-Wan Kwon as third author. The second paper does not have Young-Wan Kwon as an author, and has four additional authors with various affiliations. The second paper appears to have been was prepared in LaTeX, and the first paper appears to have been prepared in Word. The title and abstract of the first paper explicitly claim the creation of the world's first room temperature and pressure superconductor. The title and abstract of the second paper don't explicitly claim demonstration of the first superconductor, though they use some terminology that sounds like superconducting properties. The accusation is that Young-Wan Kwon published the first paper without the consent of the rest of the LK-99 team, listed himself as third author, and left off the other four. Two hours later, the rest of the LK-99 team stuffed as much as they had into the second paper, and released it as soon as possible. To me that totally seems like what happened. It explains why there are two different papers from the same group submitted on the same subject on the same day, and it explains why the author lists are different between the two. I haven't yet looked in detail, but I'm betting it also explains a lot of the oddities that people have noted in the first two papers. This also makes me way more excited about the possibility these claims are legit. The information so far is consistent with a research group that was forced to publish early, and who produced a superconductor through a fabrication process that is a bit tricky. There's nowhere near enough evidence to conclude LK-99 is a room temperature superconductor. But one failed replication doesn't prove LK-99 isn't a superconductor - if the fabrication process is finicky we'd expect to see a few dozen failed reproductions and a few successful reproductions [1] https://arxiv.org/abs/2307.12008 https://arxiv.org/abs/2307.12008 [2] https://arxiv.org/abs/2307.12037 https://arxiv.org/abs/2307.12037

deleted 3y ago

[deleted]

bruce511 3y ago

It's not dissimilar to the way food recipe books work. Creating a dish, replicating the dish, writing a recipe, getting competent cooks to follow the recipe, getting non-competant cooks to follow it, and so on. [1] It all leads to multiple revisions and takes a lot of time. And this is for -food-. Imagine doing it for something a lot less tolerant of variation. [1] old recipes start with "pluck 2 chickens" because, you know, someone followed the recipe and didn't pluck the chickens...

comboy 3y ago

This is a good summary: https://youtu.be/RjzL9cS3VW8?t=32 https://youtu.be/RjzL9cS3VW8?t=32 TLDW it seems very unlikely that they have what they claim

lucubratory 3y ago
In Sabine's opinion. I've seen Sabine's video when it was posted and intend no disrespect to her. The predominant theoretical explanation for the alleged new superconductor is that it may be a new type of superconductor, not type I or type II, or it may be presenting in unusual ways. There have been a bunch of confusing theoretical prodictions, like anisotropic conduction (how would that affect the way levitation looks)?, chance-based copper substitution in the lattice that could lead to superconducting clusters in what is otherwise just normal lead apatite (how would that affect the way levitation looks in a bulk chunk like that?), etc. Plenty of people who know the theory of superconductors are sceptical. They're not wrong about the theory. It's also worth pointing out that every previous time a new superconductor critical temp limit was broken in a major way, superconductor theory saying it shouldn't be possible was a cause for early scepticism amongst those who knew the theory. Sabine's gut reactions in a YouTube video doesn't make the result very unlikely. It doesn't make her wrong either, she's right that all existing superconductors, if pure and homogeneous, would not behave like that on a magnet. It is also the case that all existing superconductors do not superconduct at room temperature and ambient pressure. It is also the case that (absent fraud, which is on everyone's mind after Ranga Dias) we don't know of any material that could plausible behave the way that video shows something behaving. The only thing I can think of that could behave that way would be impure/heterogeneous diamagnetic material, but like 15x as diamagnetic as the next highest non-superconductor, which means if that material exists we don't know about it. If it was fraud, it would be with a magnetised sample on a magnet like that. The reason I think fraud is the only option if it was a magnetised sample is because that's a really, really obvious explanation for the effect and they have to have measured the sample for magnetism before putting it on a magnet. They would also literally feel it/see it/hear it while placing the sample on the magnet. Also it would almost certainly have to be composed of different elements than they describe, because lead apatite with copper definitely should not be ferromagnetic at all.

michaelt 3y ago

> Combined with the slipshod methodology, doesn't this make you feel like it was silly to release the arXiv paper when they did? Well, the LK-99 team agree As I understand it, the 99 in the name is because they first discovered it in 1999. Seems to me that if you didn't manage to nail down the process in 24 years, it's probably time to publish and let someone else have a go.

londons_explore 3y ago
And if this technology is on the critical path to solving world hunger/curing cancer or ending wars, then those things have been delayed 24 years, and millions of people are dead, all so these guys could have a shot at a paper in Nature. Humans really are selfish sometimes.
- simmerup 3y ago
  Assuming the worst of people is bad for your mental health
guerrilla 3y ago
It sounds like you don't know the history. They were doing other unrelated things in the meantime, not working in this.
- michaelt 3y ago
  I'm aware of that. But if I was an early-career researcher looking to establish myself, secure funding and maybe a tenure-track position, and I had a nobel-prize-winning discovery up my sleeve, I can't imagine many things that would take precedence. I mean, obviously whatever happened happened. It just seems strange to simultaneously claim to have discovered something 24 years ago, and to have been rushed into publishing prematurely.
  - guerrilla 3y ago
    > It just seems strange to simultaneously claim to have discovered something 24 years ago, and to have been rushed into publishing prematurely. They had to get other jobs to survive. Somone else forced their hand with the publication. There's nothing incosistent about that.

DannyBee 3y ago

The thing is - there is a totally easy and simple mechanism to create early-stage believability. Take a video of you producing and testing it, with a mobile phone. It only works 10% of the time? Not an issue, post 10 videos. Whatever. Would it be perfect? No, of course it could be faked. But i guaranteed people would be a ton less skeptical and willing to push hard on this if the original authors had done this sort of thing. It's honestly not that hard to set up a tripod and a mobile phone, and even in the circumstances here, where they apparently got pushed to publish quickly, it could still show that it works. It also would help those trying to reproduce see exactly what you did, in detail.

bluGill 3y ago
Production takes more than a day - I don't know of any phone that can take videos lasting that long (if people are even willing to watch it). You can of course take several videos and splice them together. Worse yet, I can with a bit of trick photography and standard liquid nitrogen cooled super conductors make a video that looks like success even though I'm working with colored wax for materials in the production process. (liquid nitrogen is hard to disguise, but with some trick photography and multiple takes I can do it). Or I can skip all that and spend some time in blender to do the above via CGI. (well I don't know blender that well, and most blender animations are obvious, but someone with a few months could pull this off)
- DannyBee 3y ago
  Ignore mobile phones, put in a nest, whatever. It is not that hard or expensive to record footage for a few days at a time having a continuous overwrite. You likely don't even need to go that far -Most universities have cameras everywhere, anyway, including plenty of universities that have cameras in every lab of this kind. It is also done by millions of home users all the time. As I said, yes, things can be faked, but who cares that’s not the point
mhb 3y ago
Are you suggesting that they take phone video of all their experiments over the course of multiple years? Or just the one they know will succeed?

throwawaymaths 3y ago

we used to joke in the chemistry lab that someone once tried to reproduce a reaction that was done in india. Procedure called for boiling ether, room temperature, WTF. Of course, room temperature in india can get up to 40C.