12 ms·

I find it really cool that this whole process didn't involve scientific publishers at all. Paper was submitted to Arxiv, there was Turmoil. Now a national lab i

I find it really cool that this whole process didn't involve scientific publishers at all. Paper was submitted to Arxiv, there was Turmoil. Now a national lab is reviewing it, all without a Journal submission.

xorbax 3y ago
I don't think LK99 is a superconductor But as an experimental chemist who got a PhD and works at a national lab, if the twitch/Twitter streamers don't produce a semiconductor, it says nothing Experiments and processes are hard and particular. You're expecting to produce a nanoscale material through macroacale processes. Chemistry can be like producing a microchip with a ballpeen hammer. "I hit it seventy times in thirty seconds in 3.8mL acetone and it just forms a single crystal structure. Do not hit it seventy-one or sixty-nine times." Chemists are experts at clearly explaining their procedure while leaving out the meaningless keystone detail that will only take a year or two to suss out if you're a clever experimentalist. And this isn't a clearly outlined procedure. But, again, I think the best case scenario is to watch an ambitious amateur do materials synthesis and see it work less perfectly than he intended. Maybe he and they stick around for the other 5 or 10 or 150 weeks as he does the same thing again to make sure the problem is nature rather than technique. If bet-placing is the fashion, shall we place bets about tenacity and thoroughness?
- wahern 3y ago
  I wonder if there's a language barrier. I can't judge for myself--neither the substantive content nor the biographies--but hypothetically the principle investigators might not have strong (or any) English language proficiency, relying on a younger, less experienced peer to write the paper. As someone who sucks at languages, I can appreciate that learning new languages does not come easy to everybody, or even to most people, especially in older age. If it seems like all reputable scientists have some English proficiency, that's a selection effect--within the English-speaking world in particular you're not going to bump into or even hear much about those who cannot speak or write English. AFAIU there's a substantial amount of high quality research in, e.g., Japan that is invisible to non-Japanese speakers. Japan's scientific establishment long predates English as a global language, unlike China; and unlike many European countries but similar to the United States there was never a widespread norm of learning foreign languages even among the professional classes, AFAIU. Though Korea is a younger industrialized country, I wouldn't take for granted that the principal discoverers have any English proficiency, especially considering that they would be late middle-age or older if they were working in the lab in 1999.
  - porknubbins 3y ago
    There might be a body of historical purely Japanese scientific research, but I know some Japanese physicists who all speak English well enough, and have heard others on podcasts who do also. It would be serevely limiting to not speak English when that is the language of modern science. I don’t know how far back you would have to go to find monolingual scientists but I would guess several decades at least.
    - xorbax 3y ago
      I find it sort of implausible that somebody high in their field is totally ignorant in English, if only because so many papers are only published in English. I've known so many researchers who apologize every other sentence because they say they've only read English and I would never had known otherwise. Which is to say - it'd be wierd to say "well I don't know English and I don't know anybody who does oh well". But, anyway, the papers were published on aeXive in...
    - rtpg 3y ago
      There are plenty of Japanese scientists who are _at least_ extremely uncomfortable in English. Plenty of people who struggle to write their own stuff in English and get it corrected a lot. Though I think that beyond a certain level you have to be interacting internationally to get anywhere in your career so you're gonna be forced along. Plenty of people would learn Latin to consume some content, but doesn't mean they are proficient in it itself, would be another example. Having said that, I think that in the research field that merely ends up with "very confusingly written papers with mistakes large enough to hide process errors" more than "cannot communicate any info in English". But there are plenty of people who get their doctorate and their only English-language output is their final papers in themselves.
    - Sharlin 3y ago
      You realize there’s an incredibly obvious selection bias in your sample?
      - porknubbins 3y ago
- TheAceOfHearts 3y ago
  Something I've also been thinking about is that we don't appear to have any data on the success rate. Maybe the samples of LK99 are the best that the team has been able to produce, but the process requires further refinement and experimentation.
  - xorbax 3y ago
    It's a little strange that they don't do multiple grinding/crystallization when their best sample produces...gimpy diamagnetism If it's been 1000 experiments, do another. I'm not convinced that 1000 experiments over 24 years is all that intense if you actually do the math.
    - cryptonector 3y ago
      1000 / 24y ~= 41/y, which after vacation and other things is probably closer to 1/wk. If all their experiments involve easy-to-synthesize substances (where easy == no more than 3 or 4 days) then it's not that intense, no. But you really have to be quite confident that this approach will pay off to put 2+ decades into it. I suspect if they really did 1,000 experiments then they often did 2/wk because I don't think they ran all of them continuously over 20+yrs. Speaking of which, can we see their notes on the other 999 experiments that didn't work out? That would be very useful, especially if LK-99 turns out not to be awesome but is good enough to hint that this approach is worthy of more research -- there's 999 compounds not worth testing, right? (Well, maybe maybe. If the physics of LK-99 come to be understood, then maybe some of the other 999 experiments might need to be re-examined.)
      - xorbax 3y ago
        This is part of my...curiosity. They didn't have a theory to drive them to doggedly pursue this system as an SC. Was it just the blip 20 years ago? Why wouldn't you wonder if your machine was shitty? If I was them, its decades past believing this might be an empty rabbit hole. You're committed. If it's real, it's heroic. If not? It's a failure of rationality. But it's both simultaneously. If they're right, I'm sure every notebook will be published. If not? We'll never see their fury or process.
- iancmceachern 3y ago
  Thank you for this, especially the hammer analogy. For a humble mechanical engineer its very helpful to make hammer analogies.
  - xorbax 3y ago
    Chemistry is a wierd field It's too complex to be perfectly predictable, like particle physics But without instruments,you have to hope the nanoscale changes are slow and visible in solution at 1,000,000x larger scale The fact that we can throw shit in a beaker and identify useful reactions is a hell of a thing. Our magic balls are the only thing that take it much beyond it's origins in alchemy.
- wrp 3y ago
  My dad was an Organiker and he used to go on about the huge amount of lab-craft knowledge you needed beyond what is communicated in reports. One of his favorite movies was It Happens Every Spring (1949), about a chemist who produces a non-reproducible compound.
  - xorbax 3y ago
    I'll have to give that movie a watch. A huge part of science is being a knowledgeable and rigorous experimentalist. You can do every great for years, but you still have to do things right 25 years in or you'll be unable to sleep
- swader999 3y ago
  Breaking bad showed us all how hard chemistry is. Still rooting for the after hours twitch/Twitter hackers.
- emtel 3y ago
  > if the twitch/Twitter streamers don't produce a semiconductor, it says nothing If you're referring to Andrew McCalip, he's been very clear that he is not claiming to be doing a serious replication attempt, but he also does zero-g/orbital chemical synthesis for a living so its not quite fair to characterize him as a random twitch streamer.
godelski 3y ago
As an academic, I have not so secretly been enjoying the shit storm of revelations lately. Science got along fine for thousands of years without Journals and Conferences deciding what's "correct" and not. It's been a failed experiment. Journals started as a way to help distribute works to other researchers to improve upon the old method of just mailing it to one another. Arxiv is great for peer review. Your work is actually fucking available and not behind some pay wall, where your peers can... review it. Only thing would be better is if we were using OpenReview so we could track discussions, but I'll admit that could get messy real quick as anyone that's open sourced their research work will tell you (lots of questions like "I trained my model, how do I test it?" and "I'm getting a cuda out of memory error, how do I fix this?"). Arxiv/preprints are peer review
- xorbax 3y ago
  The thing that established journals do better than arXiv is clarity You can upload whatever trash methods you want, but a normal journal will have at least one guy who tells you to wipe your ass and make your bullshit presentable in public, if only because they're expected to be gatekeepers for a minimum standard of supposed reproducibility.
  - godelski 3y ago
    In practice I'm not seeing much of a difference. Maybe it is just being in ML, where if you wait for conferences you're far behind. If a paper is that shitty, it is usually very apparent. Like if a paper isn't in latex you know... I mean there's a lot of garbage in conferences and journals too, I just haven't found it to be a meaningful signal. And it is still silly that people call it "peer review." Peer review is not 3-4 randos briefly glancing my work in an adversarial setting who say my work is not novel because it is the same as some unrelated work that they didn't read either; peer review is the grad student building on top of my work, peer review is lucidrains rewriting my work from scratch, it is Ross Wightman integrating it into timm and retraining, it is the forks that use my work for projects (hobby or professional). Peer review is peers looking and reviewing. More peer review happens on Twitter than these conferences. You can say these conferences and journals are a form of peer review, but we gotta stop saying that just because something is a preprint that it isn't peer reviewed. That's just incorrect. Peer review is when peers review.
systemvoltage 3y ago
Somewhere I read that this is how things used to be in Einstein era (or maybe earlier than that). There was little bureaucracy in getting the idea out.
- ggm 3y ago
  Richard Rhodes' monumental work on the A and H bomb mentions that from Thompson onward, "being heard" Was a problem. Juniors often struggled to be taken seriously, sometimes seniors ostentatiously "spoke for them" to have new ideas given credence and women were doubly disabled on the "taken seriously" front. Right up until Murray Gell-Mann and beyond, speaking outside the current limitations to knowledge was hard. I don't want this to descent into AGW and antivaxx denialism, this is inside classic science but considering radical theory/paradigm shifts which were still testable propositions. New models are hard on early stage career. Blinded peer review was partly designed to help some of that. In a narrow enough field it's impossible for reviewer and submittor not to know each other. There may only be 3-4 people who understand your niche fully. Rueben Hersh discusses that a bit in "what is mathematics really" (I think, could be another book of his, "the mathematical experience") Rhodes discusses Michael Polyani's theory of science as old fashioned apprenticeship. Journeyman scientists publish reproducible, testable work. Theoreticians.. harder to test sometimes.
  - nextos 3y ago
    I agree with what you say but the problem is that journals such as Nature do not have blinded peer review. Reviewers know who you are. This is quite shocking to discover if you come from Math or CS.
    - ggm 3y ago
      Yes. The system broke down. I don't know anyone who thinks it works how it should I've had pretty hard bounces which were deserved and I know how to get work over the threshold, but some review feedback has been petty, passive aggressive ignorance, and suspiciously similar stuff pops up from time to time in "3 papers and your phd is done" which makes me wonder if copycats are getting softballed through for career development. the rules behind length, word count and mark-up are pretty silly too. I've seen some rather odd latex tricks to compensate for length inside word count because typographical checkers were bouncing. I don't do this for a living, barely act as maybe helpful co author these days: it's hard work.
lloydatkinson 3y ago
Why should experiments wait to be submitted to some arbitrary academic body before trying to to reproduce them? This is what’s wrong with modern science, everything is academic.
- kykeonaut 3y ago
  The only thing valuable that academic journals and conferences bring to the table is the peer-review process. Which, to be fair, is needed. In this case, the peer review is being conducted without the need for publishing venues. However, I don't think that is usually the case. Take a look at the countless number of ArXiv papers that lack any sort of peer review.
oldgradstudent 3y ago
> I find it really cool that this whole process didn't involve scientific publishers at all. IIRC, the 1989 cold fusion fiasco did not involve scientific publishers as well. Not saying this is a similar fiasco, just that 34 years ago it was done via a press conference and circulating drafts rather than arXiv.
daniel-cussen 3y ago
[dead]

xorbax 3y ago

I don't think LK99 is a superconductor But as an experimental chemist who got a PhD and works at a national lab, if the twitch/Twitter streamers don't produce a semiconductor, it says nothing Experiments and processes are hard and particular. You're expecting to produce a nanoscale material through macroacale processes. Chemistry can be like producing a microchip with a ballpeen hammer. "I hit it seventy times in thirty seconds in 3.8mL acetone and it just forms a single crystal structure. Do not hit it seventy-one or sixty-nine times." Chemists are experts at clearly explaining their procedure while leaving out the meaningless keystone detail that will only take a year or two to suss out if you're a clever experimentalist. And this isn't a clearly outlined procedure. But, again, I think the best case scenario is to watch an ambitious amateur do materials synthesis and see it work less perfectly than he intended. Maybe he and they stick around for the other 5 or 10 or 150 weeks as he does the same thing again to make sure the problem is nature rather than technique. If bet-placing is the fashion, shall we place bets about tenacity and thoroughness?

wahern 3y ago
I wonder if there's a language barrier. I can't judge for myself--neither the substantive content nor the biographies--but hypothetically the principle investigators might not have strong (or any) English language proficiency, relying on a younger, less experienced peer to write the paper. As someone who sucks at languages, I can appreciate that learning new languages does not come easy to everybody, or even to most people, especially in older age. If it seems like all reputable scientists have some English proficiency, that's a selection effect--within the English-speaking world in particular you're not going to bump into or even hear much about those who cannot speak or write English. AFAIU there's a substantial amount of high quality research in, e.g., Japan that is invisible to non-Japanese speakers. Japan's scientific establishment long predates English as a global language, unlike China; and unlike many European countries but similar to the United States there was never a widespread norm of learning foreign languages even among the professional classes, AFAIU. Though Korea is a younger industrialized country, I wouldn't take for granted that the principal discoverers have any English proficiency, especially considering that they would be late middle-age or older if they were working in the lab in 1999.
- porknubbins 3y ago
  There might be a body of historical purely Japanese scientific research, but I know some Japanese physicists who all speak English well enough, and have heard others on podcasts who do also. It would be serevely limiting to not speak English when that is the language of modern science. I don’t know how far back you would have to go to find monolingual scientists but I would guess several decades at least.
  - xorbax 3y ago
    I find it sort of implausible that somebody high in their field is totally ignorant in English, if only because so many papers are only published in English. I've known so many researchers who apologize every other sentence because they say they've only read English and I would never had known otherwise. Which is to say - it'd be wierd to say "well I don't know English and I don't know anybody who does oh well". But, anyway, the papers were published on aeXive in...
  - rtpg 3y ago
    There are plenty of Japanese scientists who are _at least_ extremely uncomfortable in English. Plenty of people who struggle to write their own stuff in English and get it corrected a lot. Though I think that beyond a certain level you have to be interacting internationally to get anywhere in your career so you're gonna be forced along. Plenty of people would learn Latin to consume some content, but doesn't mean they are proficient in it itself, would be another example. Having said that, I think that in the research field that merely ends up with "very confusingly written papers with mistakes large enough to hide process errors" more than "cannot communicate any info in English". But there are plenty of people who get their doctorate and their only English-language output is their final papers in themselves.
  - Sharlin 3y ago
    You realize there’s an incredibly obvious selection bias in your sample?
    - porknubbins 3y ago
TheAceOfHearts 3y ago
Something I've also been thinking about is that we don't appear to have any data on the success rate. Maybe the samples of LK99 are the best that the team has been able to produce, but the process requires further refinement and experimentation.
- xorbax 3y ago
  It's a little strange that they don't do multiple grinding/crystallization when their best sample produces...gimpy diamagnetism If it's been 1000 experiments, do another. I'm not convinced that 1000 experiments over 24 years is all that intense if you actually do the math.
  - cryptonector 3y ago
    1000 / 24y ~= 41/y, which after vacation and other things is probably closer to 1/wk. If all their experiments involve easy-to-synthesize substances (where easy == no more than 3 or 4 days) then it's not that intense, no. But you really have to be quite confident that this approach will pay off to put 2+ decades into it. I suspect if they really did 1,000 experiments then they often did 2/wk because I don't think they ran all of them continuously over 20+yrs. Speaking of which, can we see their notes on the other 999 experiments that didn't work out? That would be very useful, especially if LK-99 turns out not to be awesome but is good enough to hint that this approach is worthy of more research -- there's 999 compounds not worth testing, right? (Well, maybe maybe. If the physics of LK-99 come to be understood, then maybe some of the other 999 experiments might need to be re-examined.)
    - xorbax 3y ago
      This is part of my...curiosity. They didn't have a theory to drive them to doggedly pursue this system as an SC. Was it just the blip 20 years ago? Why wouldn't you wonder if your machine was shitty? If I was them, its decades past believing this might be an empty rabbit hole. You're committed. If it's real, it's heroic. If not? It's a failure of rationality. But it's both simultaneously. If they're right, I'm sure every notebook will be published. If not? We'll never see their fury or process.
iancmceachern 3y ago
Thank you for this, especially the hammer analogy. For a humble mechanical engineer its very helpful to make hammer analogies.
- xorbax 3y ago
  Chemistry is a wierd field It's too complex to be perfectly predictable, like particle physics But without instruments,you have to hope the nanoscale changes are slow and visible in solution at 1,000,000x larger scale The fact that we can throw shit in a beaker and identify useful reactions is a hell of a thing. Our magic balls are the only thing that take it much beyond it's origins in alchemy.
wrp 3y ago
My dad was an Organiker and he used to go on about the huge amount of lab-craft knowledge you needed beyond what is communicated in reports. One of his favorite movies was It Happens Every Spring (1949), about a chemist who produces a non-reproducible compound.
- xorbax 3y ago
  I'll have to give that movie a watch. A huge part of science is being a knowledgeable and rigorous experimentalist. You can do every great for years, but you still have to do things right 25 years in or you'll be unable to sleep
swader999 3y ago
Breaking bad showed us all how hard chemistry is. Still rooting for the after hours twitch/Twitter hackers.
emtel 3y ago
> if the twitch/Twitter streamers don't produce a semiconductor, it says nothing If you're referring to Andrew McCalip, he's been very clear that he is not claiming to be doing a serious replication attempt, but he also does zero-g/orbital chemical synthesis for a living so its not quite fair to characterize him as a random twitch streamer.

godelski 3y ago

As an academic, I have not so secretly been enjoying the shit storm of revelations lately. Science got along fine for thousands of years without Journals and Conferences deciding what's "correct" and not. It's been a failed experiment. Journals started as a way to help distribute works to other researchers to improve upon the old method of just mailing it to one another. Arxiv is great for peer review. Your work is actually fucking available and not behind some pay wall, where your peers can... review it. Only thing would be better is if we were using OpenReview so we could track discussions, but I'll admit that could get messy real quick as anyone that's open sourced their research work will tell you (lots of questions like "I trained my model, how do I test it?" and "I'm getting a cuda out of memory error, how do I fix this?"). Arxiv/preprints are peer review

xorbax 3y ago
The thing that established journals do better than arXiv is clarity You can upload whatever trash methods you want, but a normal journal will have at least one guy who tells you to wipe your ass and make your bullshit presentable in public, if only because they're expected to be gatekeepers for a minimum standard of supposed reproducibility.
- godelski 3y ago
  In practice I'm not seeing much of a difference. Maybe it is just being in ML, where if you wait for conferences you're far behind. If a paper is that shitty, it is usually very apparent. Like if a paper isn't in latex you know... I mean there's a lot of garbage in conferences and journals too, I just haven't found it to be a meaningful signal. And it is still silly that people call it "peer review." Peer review is not 3-4 randos briefly glancing my work in an adversarial setting who say my work is not novel because it is the same as some unrelated work that they didn't read either; peer review is the grad student building on top of my work, peer review is lucidrains rewriting my work from scratch, it is Ross Wightman integrating it into timm and retraining, it is the forks that use my work for projects (hobby or professional). Peer review is peers looking and reviewing. More peer review happens on Twitter than these conferences. You can say these conferences and journals are a form of peer review, but we gotta stop saying that just because something is a preprint that it isn't peer reviewed. That's just incorrect. Peer review is when peers review.

systemvoltage 3y ago

Somewhere I read that this is how things used to be in Einstein era (or maybe earlier than that). There was little bureaucracy in getting the idea out.

ggm 3y ago
Richard Rhodes' monumental work on the A and H bomb mentions that from Thompson onward, "being heard" Was a problem. Juniors often struggled to be taken seriously, sometimes seniors ostentatiously "spoke for them" to have new ideas given credence and women were doubly disabled on the "taken seriously" front. Right up until Murray Gell-Mann and beyond, speaking outside the current limitations to knowledge was hard. I don't want this to descent into AGW and antivaxx denialism, this is inside classic science but considering radical theory/paradigm shifts which were still testable propositions. New models are hard on early stage career. Blinded peer review was partly designed to help some of that. In a narrow enough field it's impossible for reviewer and submittor not to know each other. There may only be 3-4 people who understand your niche fully. Rueben Hersh discusses that a bit in "what is mathematics really" (I think, could be another book of his, "the mathematical experience") Rhodes discusses Michael Polyani's theory of science as old fashioned apprenticeship. Journeyman scientists publish reproducible, testable work. Theoreticians.. harder to test sometimes.
- nextos 3y ago
  I agree with what you say but the problem is that journals such as Nature do not have blinded peer review. Reviewers know who you are. This is quite shocking to discover if you come from Math or CS.
  - ggm 3y ago
    Yes. The system broke down. I don't know anyone who thinks it works how it should I've had pretty hard bounces which were deserved and I know how to get work over the threshold, but some review feedback has been petty, passive aggressive ignorance, and suspiciously similar stuff pops up from time to time in "3 papers and your phd is done" which makes me wonder if copycats are getting softballed through for career development. the rules behind length, word count and mark-up are pretty silly too. I've seen some rather odd latex tricks to compensate for length inside word count because typographical checkers were bouncing. I don't do this for a living, barely act as maybe helpful co author these days: it's hard work.

lloydatkinson 3y ago

Why should experiments wait to be submitted to some arbitrary academic body before trying to to reproduce them? This is what’s wrong with modern science, everything is academic.

kykeonaut 3y ago
The only thing valuable that academic journals and conferences bring to the table is the peer-review process. Which, to be fair, is needed. In this case, the peer review is being conducted without the need for publishing venues. However, I don't think that is usually the case. Take a look at the countless number of ArXiv papers that lack any sort of peer review.

oldgradstudent 3y ago

> I find it really cool that this whole process didn't involve scientific publishers at all. IIRC, the 1989 cold fusion fiasco did not involve scientific publishers as well. Not saying this is a similar fiasco, just that 34 years ago it was done via a press conference and circulating drafts rather than arXiv.

daniel-cussen 3y ago

[dead]