4 ms·

The simple truth is we don’t have a culture of replication because no one is paying and no one is paying because no one is demanding it as part of the incentive

The simple truth is we don’t have a culture of replication because no one is paying and no one is paying because no one is demanding it as part of the incentive system that the money is part of… If a prerequisite of publication in a top tier journal was someone having replicated the work… anyone at all… then the quality of write up would improve dramatically because all the stuff that never gets written up would be written up so that the second lab costs them less money to do the replication work, so the second lab doesn’t spent a month failing because they didn’t bake the powder in a vacuum for a week before performing the entire synthesis under dry nitrogen purge when the methodology just said the powder had been dried and the synthesis was performed at ambient pressure in a zero humidity atmosphere…

grall 3y ago
> If a prerequisite of publication in a top tier journal was someone having replicated the work… anyone at all For a SciFi take on this see the section "A Note on Governance" at https://eldraeverse.com/2018/05/25/approach-vector-2-2/ https://eldraeverse.com/2018/05/25/approach-vector-2-2/. Specifically, the "Dean of Evidence".
Someone 3y ago
> If a prerequisite of publication in a top tier journal was someone having replicated the work… anyone at all… then the quality of write up would improve dramatically I would expect pairs of labs to gang up, with one of them giving the other the details so that they can reproduce the result, and the contents of the publication staying more or less the same. Reason is that everybody wants to prevent others from being able to build on one’s work faster than one can do themselves. Even if the lab you give more information isn’t run by a buddy of you, having a single competitor is better than having the whole world compete with you. For example, in this case, people will start cooking all kinds of variations on this stuff to find one that’s easier to produce, allows for higher currents before losing superconductivity, doesn’t have lead in it, etc.
- techdragon 3y ago
  Oh absolutely, there’s a lot of potential issues with the simple suggestion of requiring replication. I was tempted to write a bunch more to outline strategies and counter strategies from a game theory perspective but realised it was basically unnecessary since it will never happen.
pjc50 3y ago
> The simple truth is we don’t have a culture of replication because no one is paying and no one is paying because no one is demanding it as part of the incentive system that the money is part of… A darker comment: the non-replication of results implies that nobody's actually making use of the science output, and an alarmingly large fraction of unreplicated positive results are simply a waste of time.

grall 3y ago

> If a prerequisite of publication in a top tier journal was someone having replicated the work… anyone at all For a SciFi take on this see the section "A Note on Governance" at https://eldraeverse.com/2018/05/25/approach-vector-2-2/ https://eldraeverse.com/2018/05/25/approach-vector-2-2/. Specifically, the "Dean of Evidence".

Someone 3y ago

> If a prerequisite of publication in a top tier journal was someone having replicated the work… anyone at all… then the quality of write up would improve dramatically I would expect pairs of labs to gang up, with one of them giving the other the details so that they can reproduce the result, and the contents of the publication staying more or less the same. Reason is that everybody wants to prevent others from being able to build on one’s work faster than one can do themselves. Even if the lab you give more information isn’t run by a buddy of you, having a single competitor is better than having the whole world compete with you. For example, in this case, people will start cooking all kinds of variations on this stuff to find one that’s easier to produce, allows for higher currents before losing superconductivity, doesn’t have lead in it, etc.

techdragon 3y ago
Oh absolutely, there’s a lot of potential issues with the simple suggestion of requiring replication. I was tempted to write a bunch more to outline strategies and counter strategies from a game theory perspective but realised it was basically unnecessary since it will never happen.

pjc50 3y ago

> The simple truth is we don’t have a culture of replication because no one is paying and no one is paying because no one is demanding it as part of the incentive system that the money is part of… A darker comment: the non-replication of results implies that nobody's actually making use of the science output, and an alarmingly large fraction of unreplicated positive results are simply a waste of time.