4 ms·

We also don't have a culture of replication. Basically all of the post docs I worked with in the lab lamented how hard it can be to replicate reported syntheses

We also don't have a culture of replication. Basically all of the post docs I worked with in the lab lamented how hard it can be to replicate reported syntheses in chemistry wet labs. Everything from humidity and altitude to undocumented "magic tricks" (eg baking something in an oven under vaccuum for a week to get.rid.of.all.the.water.) I heard anecdotes that experiments were sometimes literally only reproducible in one exact lab because reasons. Turns out there's just as much art as there is science in experimentation...

mschuster91 3y ago
The problem IMHO is funding related. Often enough the original lab's grant is barely enough to account for the research and a shoddily put together paper, the rest is filled with a crapload of un(der)paid/"voluntold" labour. It's rare for researchers to get the budget to replicate someone else's work - this case might be one of the few exceptions given that everyone and their dog want to be the first one to show the world a successful replication. Hell, there are Youtube and Tiktok streamers on the task as well, that way you know just how nuts everyone is going.
- michaelt 3y ago
  Even in well funded fields, nobody wants to give away easy follow-on publications to other labs. Oh, you want to document things well enough that people can build on your work and give you citations eventually. But if you've just demonstrated a groundbreaking technique for Foo and the obvious next step is to test it on Bar, you've got an opportunity to get your lab two papers instead of one, and all you have to do is less documentation?
techdragon 3y ago
The simple truth is we don’t have a culture of replication because no one is paying and no one is paying because no one is demanding it as part of the incentive system that the money is part of… If a prerequisite of publication in a top tier journal was someone having replicated the work… anyone at all… then the quality of write up would improve dramatically because all the stuff that never gets written up would be written up so that the second lab costs them less money to do the replication work, so the second lab doesn’t spent a month failing because they didn’t bake the powder in a vacuum for a week before performing the entire synthesis under dry nitrogen purge when the methodology just said the powder had been dried and the synthesis was performed at ambient pressure in a zero humidity atmosphere…
- grall 3y ago
  > If a prerequisite of publication in a top tier journal was someone having replicated the work… anyone at all For a SciFi take on this see the section "A Note on Governance" at https://eldraeverse.com/2018/05/25/approach-vector-2-2/ https://eldraeverse.com/2018/05/25/approach-vector-2-2/. Specifically, the "Dean of Evidence".
- Someone 3y ago
  > If a prerequisite of publication in a top tier journal was someone having replicated the work… anyone at all… then the quality of write up would improve dramatically I would expect pairs of labs to gang up, with one of them giving the other the details so that they can reproduce the result, and the contents of the publication staying more or less the same. Reason is that everybody wants to prevent others from being able to build on one’s work faster than one can do themselves. Even if the lab you give more information isn’t run by a buddy of you, having a single competitor is better than having the whole world compete with you. For example, in this case, people will start cooking all kinds of variations on this stuff to find one that’s easier to produce, allows for higher currents before losing superconductivity, doesn’t have lead in it, etc.
  - techdragon 3y ago
    Oh absolutely, there’s a lot of potential issues with the simple suggestion of requiring replication. I was tempted to write a bunch more to outline strategies and counter strategies from a game theory perspective but realised it was basically unnecessary since it will never happen.
- pjc50 3y ago
  > The simple truth is we don’t have a culture of replication because no one is paying and no one is paying because no one is demanding it as part of the incentive system that the money is part of… A darker comment: the non-replication of results implies that nobody's actually making use of the science output, and an alarmingly large fraction of unreplicated positive results are simply a waste of time.
arlcode 3y ago
In that light, industrial processes seem even more impressive. The super power of modern technology is the ability to perform precise steps on a massive scale and basically forever. Obviously it takes a lot of effort to keep the machinery running but still, the mere fact that it's running in the first place blows your mind if you think about it. Modern Machinery multiplies the impact on the world of any human who operates it.
- djtango 3y ago
  Research experimentation is like doing a gymnastics performance, you're trying to prove the limits of what can be done. Industry is all about doing it cheap, consistently and at scale. So by comparison you don't want any flourish or special conditions that take lots of practice and super skilled operators. The magic of industrial engineering is figuring out how to go from research to scale. But the flipside is all the crazy stuff that is achievable at the frontier that never gets productionised
ilc 3y ago
Look up FOGBANK (nuclear material), if you want to see what happens when people get REALLY motivated to replicate the impossible to replicate.
kitd 3y ago
A chemical well-it-works-on-my-computer experiment. Nice!
whimsicalism 3y ago
This was my experience as well, my job was basically to specialize in reverse engineering those exact “baking” techniques in the tube furnace and it is so challenging and subject to variability.
paulmd 3y ago
> baking something in an oven under vaccuum for a week to get.rid.of.all.the.water. vacuum oven also works great for drying 3d printer filament!

mschuster91 3y ago

The problem IMHO is funding related. Often enough the original lab's grant is barely enough to account for the research and a shoddily put together paper, the rest is filled with a crapload of un(der)paid/"voluntold" labour. It's rare for researchers to get the budget to replicate someone else's work - this case might be one of the few exceptions given that everyone and their dog want to be the first one to show the world a successful replication. Hell, there are Youtube and Tiktok streamers on the task as well, that way you know just how nuts everyone is going.

michaelt 3y ago
Even in well funded fields, nobody wants to give away easy follow-on publications to other labs. Oh, you want to document things well enough that people can build on your work and give you citations eventually. But if you've just demonstrated a groundbreaking technique for Foo and the obvious next step is to test it on Bar, you've got an opportunity to get your lab two papers instead of one, and all you have to do is less documentation?

techdragon 3y ago

The simple truth is we don’t have a culture of replication because no one is paying and no one is paying because no one is demanding it as part of the incentive system that the money is part of… If a prerequisite of publication in a top tier journal was someone having replicated the work… anyone at all… then the quality of write up would improve dramatically because all the stuff that never gets written up would be written up so that the second lab costs them less money to do the replication work, so the second lab doesn’t spent a month failing because they didn’t bake the powder in a vacuum for a week before performing the entire synthesis under dry nitrogen purge when the methodology just said the powder had been dried and the synthesis was performed at ambient pressure in a zero humidity atmosphere…

grall 3y ago
> If a prerequisite of publication in a top tier journal was someone having replicated the work… anyone at all For a SciFi take on this see the section "A Note on Governance" at https://eldraeverse.com/2018/05/25/approach-vector-2-2/ https://eldraeverse.com/2018/05/25/approach-vector-2-2/. Specifically, the "Dean of Evidence".
Someone 3y ago
> If a prerequisite of publication in a top tier journal was someone having replicated the work… anyone at all… then the quality of write up would improve dramatically I would expect pairs of labs to gang up, with one of them giving the other the details so that they can reproduce the result, and the contents of the publication staying more or less the same. Reason is that everybody wants to prevent others from being able to build on one’s work faster than one can do themselves. Even if the lab you give more information isn’t run by a buddy of you, having a single competitor is better than having the whole world compete with you. For example, in this case, people will start cooking all kinds of variations on this stuff to find one that’s easier to produce, allows for higher currents before losing superconductivity, doesn’t have lead in it, etc.
- techdragon 3y ago
  Oh absolutely, there’s a lot of potential issues with the simple suggestion of requiring replication. I was tempted to write a bunch more to outline strategies and counter strategies from a game theory perspective but realised it was basically unnecessary since it will never happen.
pjc50 3y ago
> The simple truth is we don’t have a culture of replication because no one is paying and no one is paying because no one is demanding it as part of the incentive system that the money is part of… A darker comment: the non-replication of results implies that nobody's actually making use of the science output, and an alarmingly large fraction of unreplicated positive results are simply a waste of time.

arlcode 3y ago

In that light, industrial processes seem even more impressive. The super power of modern technology is the ability to perform precise steps on a massive scale and basically forever. Obviously it takes a lot of effort to keep the machinery running but still, the mere fact that it's running in the first place blows your mind if you think about it. Modern Machinery multiplies the impact on the world of any human who operates it.

djtango 3y ago
Research experimentation is like doing a gymnastics performance, you're trying to prove the limits of what can be done. Industry is all about doing it cheap, consistently and at scale. So by comparison you don't want any flourish or special conditions that take lots of practice and super skilled operators. The magic of industrial engineering is figuring out how to go from research to scale. But the flipside is all the crazy stuff that is achievable at the frontier that never gets productionised

ilc 3y ago

Look up FOGBANK (nuclear material), if you want to see what happens when people get REALLY motivated to replicate the impossible to replicate.

kitd 3y ago

A chemical well-it-works-on-my-computer experiment. Nice!

whimsicalism 3y ago

This was my experience as well, my job was basically to specialize in reverse engineering those exact “baking” techniques in the tube furnace and it is so challenging and subject to variability.

paulmd 3y ago

> baking something in an oven under vaccuum for a week to get.rid.of.all.the.water. vacuum oven also works great for drying 3d printer filament!