5 ms·

Not just materials science though. I’m a longtime reader of Derek Lowe’s In the Pipeline blog, and he often emphasizes the trickiness of lab work and lab techni

Not just materials science though. I’m a longtime reader of Derek Lowe’s In the Pipeline blog, and he often emphasizes the trickiness of lab work and lab techniques. But the model we have of science from elementary and high school doesn’t give us room for the random, ephemeral things that come into play in actual experimental work. Our model is methods equal result, every time, even though jiggling your foot a little bit to get the feeling back may have somehow made your small molecule production have less binding affinity or something. And yes, eventually you figure that out and stop jiggling your foot but two years have passed.

lucubratory 3y ago
Absolutely, ultimate failures to replicate are generally multiple labs across dozens and dozens of attempts and continuous communication with the authors. And yeah, it's a thing in science in general, materials science is just particularly susceptible because of the nature of the work.
tetha 3y ago
NileRed and other Maker Channels are great to demonstrate that. In every video, there's 1 - 2 of these situations - Nile has been following some documented, established procedure which should result in a specific reaction and it looks decent to the layman - and then it just doesn't work at all. And then he re-does pretty much the same thing and it works. I can't imagine how frustrating that can be. And then you have to do it with something you don't even know works. Kind of like dealing with obscure retro computers. Does it still work? No one knows. What does it do if it works? No one knows. Can it even function or are parts missing? That's right - no one knows.
- teaearlgraycold 3y ago
  Link to NileRed video: https://www.youtube.com/watch?v=RS7gyZJg5nc https://www.youtube.com/watch?v=RS7gyZJg5nc
- MichaelZuo 3y ago
  Well to be fair it's very difficult to nail down every possible parameter to 99.99% accuracy and precision from the original successful experiment. There's a lot of randomness even in a highly controlled, well funded, modern laboratory setting, let alone less well equipped facilities.
- whimsicalism 3y ago
  It is incredibly frustrating, my experience in an experimental physics lab involved in creation of certain types of wafers for SEM characterization, there are so many things that can go wrong for certain materials. just a certain speck of dust being in the wrong place at the wrong time It is why I quit experimental physics.
- oceanplexian 3y ago
  Anyone who has dived into FDM 3D printing knows this pain well. Filament is too wet, the air is too cold, a batch of filament might behave wrong. Or everything goes 99% right and the axis of the moon and the constellations are off and it ruins your entire print after 57 hours. Then you start again and it breaks at 59.5 hours. I’m sure it’s not as hardcore as synthesizing superconductors but the real world is incredibly messy and imprecise.
- stevage 3y ago
  But Jesus in this case the payoff is immense.
Loic 3y ago
For computer science people, I always use the yield on wafers as analogy to explain how hard it can be to nail an experiment right. Usually they understand directly because this is something they can easily relate to.
- eric-hu 3y ago
  For those of us unfamiliar with the numbers, what do typical yields look like on silicon wafers?
  - junofan 3y ago
    50–97% with 90% considered “good.”
    - XorNot 3y ago
      It's also worth contextualizing percentages. 90% still means 1 in every 10 parts fails.
      - OJFord 3y ago
        It's not always outright failure, but rather different performance binning.
        Chabsff 3y ago
        But the performance binning is a direct result of careful design that goes out of its way to ensure that a subset of failures can still lead to a workable/binnable products. It's still a failure to produce a part that matches the full design.
  - crote 3y ago
    It really varies. Samsung 3nm had a yield rate well below 20% back in early 2022, and that's not unheard of while still in the development phase. To be economically viable the yield rates have to be around 80%, so mature nodes can be expected to be in that ballpark.
djtango 3y ago
We also don't have a culture of replication. Basically all of the post docs I worked with in the lab lamented how hard it can be to replicate reported syntheses in chemistry wet labs. Everything from humidity and altitude to undocumented "magic tricks" (eg baking something in an oven under vaccuum for a week to get.rid.of.all.the.water.) I heard anecdotes that experiments were sometimes literally only reproducible in one exact lab because reasons. Turns out there's just as much art as there is science in experimentation...
- mschuster91 3y ago
  The problem IMHO is funding related. Often enough the original lab's grant is barely enough to account for the research and a shoddily put together paper, the rest is filled with a crapload of un(der)paid/"voluntold" labour. It's rare for researchers to get the budget to replicate someone else's work - this case might be one of the few exceptions given that everyone and their dog want to be the first one to show the world a successful replication. Hell, there are Youtube and Tiktok streamers on the task as well, that way you know just how nuts everyone is going.
  - michaelt 3y ago
    Even in well funded fields, nobody wants to give away easy follow-on publications to other labs. Oh, you want to document things well enough that people can build on your work and give you citations eventually. But if you've just demonstrated a groundbreaking technique for Foo and the obvious next step is to test it on Bar, you've got an opportunity to get your lab two papers instead of one, and all you have to do is less documentation?
- techdragon 3y ago
  The simple truth is we don’t have a culture of replication because no one is paying and no one is paying because no one is demanding it as part of the incentive system that the money is part of… If a prerequisite of publication in a top tier journal was someone having replicated the work… anyone at all… then the quality of write up would improve dramatically because all the stuff that never gets written up would be written up so that the second lab costs them less money to do the replication work, so the second lab doesn’t spent a month failing because they didn’t bake the powder in a vacuum for a week before performing the entire synthesis under dry nitrogen purge when the methodology just said the powder had been dried and the synthesis was performed at ambient pressure in a zero humidity atmosphere…
OJFord 3y ago
> But the model we have of science from elementary and high school doesn’t give us room for the random, ephemeral things that come into play in actual experimental work. Our model is methods equal result, every time, even though jiggling your foot a little bit to get the feeling back may have somehow made your small molecule production have less binding affinity or something. That wasn't my experience, in either high school physics or undergrad electronics. I hated keeping a lab book (just wanted to 'do' & see results I suppose) but the emphasis was always on recording as much as possible to understand what's important and what happened etc. when you review later. I suppose looking through another lens you can call that an expectation of repeatability, but it's just a matter of emphasis isn't it? You know it might be hard to repeat, so you record as much as possible about the setup & process to remove variables and control what needs to be controlled (especially if you don't yet understand what that is, or the full scope).
sterlind 3y ago
FOGBANK, the classified styrofoam-like material used as a filler in US thermonuclear warheads, had a similar issue. They needed to restart production, but the new samples didn't work! It turns out an impurity present in the original batch was critical. Once they reproduced that impurity, the material acted correctly.
AaronM 3y ago
I am not involved in lab work at all, but his "Things I won't work with" posts are wonderful
deleted 3y ago
[deleted]

lucubratory 3y ago

Absolutely, ultimate failures to replicate are generally multiple labs across dozens and dozens of attempts and continuous communication with the authors. And yeah, it's a thing in science in general, materials science is just particularly susceptible because of the nature of the work.

tetha 3y ago

NileRed and other Maker Channels are great to demonstrate that. In every video, there's 1 - 2 of these situations - Nile has been following some documented, established procedure which should result in a specific reaction and it looks decent to the layman - and then it just doesn't work at all. And then he re-does pretty much the same thing and it works. I can't imagine how frustrating that can be. And then you have to do it with something you don't even know works. Kind of like dealing with obscure retro computers. Does it still work? No one knows. What does it do if it works? No one knows. Can it even function or are parts missing? That's right - no one knows.

teaearlgraycold 3y ago
Link to NileRed video: https://www.youtube.com/watch?v=RS7gyZJg5nc https://www.youtube.com/watch?v=RS7gyZJg5nc
MichaelZuo 3y ago
Well to be fair it's very difficult to nail down every possible parameter to 99.99% accuracy and precision from the original successful experiment. There's a lot of randomness even in a highly controlled, well funded, modern laboratory setting, let alone less well equipped facilities.
whimsicalism 3y ago
It is incredibly frustrating, my experience in an experimental physics lab involved in creation of certain types of wafers for SEM characterization, there are so many things that can go wrong for certain materials. just a certain speck of dust being in the wrong place at the wrong time It is why I quit experimental physics.
oceanplexian 3y ago
Anyone who has dived into FDM 3D printing knows this pain well. Filament is too wet, the air is too cold, a batch of filament might behave wrong. Or everything goes 99% right and the axis of the moon and the constellations are off and it ruins your entire print after 57 hours. Then you start again and it breaks at 59.5 hours. I’m sure it’s not as hardcore as synthesizing superconductors but the real world is incredibly messy and imprecise.
stevage 3y ago
But Jesus in this case the payoff is immense.

Loic 3y ago

For computer science people, I always use the yield on wafers as analogy to explain how hard it can be to nail an experiment right. Usually they understand directly because this is something they can easily relate to.

eric-hu 3y ago
For those of us unfamiliar with the numbers, what do typical yields look like on silicon wafers?
- junofan 3y ago
  50–97% with 90% considered “good.”
  - XorNot 3y ago
    It's also worth contextualizing percentages. 90% still means 1 in every 10 parts fails.
    - OJFord 3y ago
      It's not always outright failure, but rather different performance binning.
      - Chabsff 3y ago
        But the performance binning is a direct result of careful design that goes out of its way to ensure that a subset of failures can still lead to a workable/binnable products. It's still a failure to produce a part that matches the full design.
- crote 3y ago
  It really varies. Samsung 3nm had a yield rate well below 20% back in early 2022, and that's not unheard of while still in the development phase. To be economically viable the yield rates have to be around 80%, so mature nodes can be expected to be in that ballpark.

djtango 3y ago

We also don't have a culture of replication. Basically all of the post docs I worked with in the lab lamented how hard it can be to replicate reported syntheses in chemistry wet labs. Everything from humidity and altitude to undocumented "magic tricks" (eg baking something in an oven under vaccuum for a week to get.rid.of.all.the.water.) I heard anecdotes that experiments were sometimes literally only reproducible in one exact lab because reasons. Turns out there's just as much art as there is science in experimentation...

mschuster91 3y ago
The problem IMHO is funding related. Often enough the original lab's grant is barely enough to account for the research and a shoddily put together paper, the rest is filled with a crapload of un(der)paid/"voluntold" labour. It's rare for researchers to get the budget to replicate someone else's work - this case might be one of the few exceptions given that everyone and their dog want to be the first one to show the world a successful replication. Hell, there are Youtube and Tiktok streamers on the task as well, that way you know just how nuts everyone is going.
- michaelt 3y ago
  Even in well funded fields, nobody wants to give away easy follow-on publications to other labs. Oh, you want to document things well enough that people can build on your work and give you citations eventually. But if you've just demonstrated a groundbreaking technique for Foo and the obvious next step is to test it on Bar, you've got an opportunity to get your lab two papers instead of one, and all you have to do is less documentation?
techdragon 3y ago
The simple truth is we don’t have a culture of replication because no one is paying and no one is paying because no one is demanding it as part of the incentive system that the money is part of… If a prerequisite of publication in a top tier journal was someone having replicated the work… anyone at all… then the quality of write up would improve dramatically because all the stuff that never gets written up would be written up so that the second lab costs them less money to do the replication work, so the second lab doesn’t spent a month failing because they didn’t bake the powder in a vacuum for a week before performing the entire synthesis under dry nitrogen purge when the methodology just said the powder had been dried and the synthesis was performed at ambient pressure in a zero humidity atmosphere…

OJFord 3y ago

> But the model we have of science from elementary and high school doesn’t give us room for the random, ephemeral things that come into play in actual experimental work. Our model is methods equal result, every time, even though jiggling your foot a little bit to get the feeling back may have somehow made your small molecule production have less binding affinity or something. That wasn't my experience, in either high school physics or undergrad electronics. I hated keeping a lab book (just wanted to 'do' & see results I suppose) but the emphasis was always on recording as much as possible to understand what's important and what happened etc. when you review later. I suppose looking through another lens you can call that an expectation of repeatability, but it's just a matter of emphasis isn't it? You know it might be hard to repeat, so you record as much as possible about the setup & process to remove variables and control what needs to be controlled (especially if you don't yet understand what that is, or the full scope).

sterlind 3y ago

FOGBANK, the classified styrofoam-like material used as a filler in US thermonuclear warheads, had a similar issue. They needed to restart production, but the new samples didn't work! It turns out an impurity present in the original batch was critical. Once they reproduced that impurity, the material acted correctly.

AaronM 3y ago

I am not involved in lab work at all, but his "Things I won't work with" posts are wonderful

deleted 3y ago

[deleted]