3 ms·

This is extremely interesting. On top of keeping and publishing "negative outcomes", could we also move to actually requiring verification and validation by an

This is extremely interesting. On top of keeping and publishing "negative outcomes", could we also move to actually requiring verification and validation by another "lab" (or really, an experiment done in different conditions)?

tomcam 2y ago
I love that idea, but it would never work in practice. Some thoughts: * Funding for any experiment would have to include 100% extra because presumably every experiment done would also have to duplicate another, randomly chosen experiment. The situation would be become something akin to lawyers being required to do pro bono work. It would mean that the randomly chosen experiment to be duplicated would require a different set of skills than the primary experiment. * Assuming the above, there would be an extremely high impedance in communications between any two of these experiments because no one could really describe their experiment in a way that would allow independence recreation of it. * Smaller institutions would struggle to re-create experiments from better funded institutions. * Getting the second experiment funded would always be difficult because you probably wouldn’t be able to go to the same sources.
- naasking 2y ago
  > Funding for any experiment would have to include 100% extra because presumably every experiment done would also have to duplicate another, randomly chosen experiment. If this were a universal policy then we'd be no worse off because everyone would face the same challenges. > Smaller institutions would struggle to re-create experiments from better funded institutions. That's already the case. > Getting the second experiment funded would always be difficult because you probably wouldn’t be able to go to the same sources. I thought we were discussing how the original experiment's funding already included funding for the replication?
analog31 2y ago
My experiment was built in a small accelerator lab, plus about $400k in 1990 dollars. Technology was evolving rapidly (lasers, computers), and some of the critical gear was obsolete by the time I finished. Had I needed to secure guaranteed funding for replication, I would not have started the experiment. How it was eventually replicated could not have been known at that point. The thing that comes to mind in this thread is that rules (and rule makers) for small biological and behavioral studies might not make sense for a physics research program, and vice versa.

tomcam 2y ago

I love that idea, but it would never work in practice. Some thoughts: * Funding for any experiment would have to include 100% extra because presumably every experiment done would also have to duplicate another, randomly chosen experiment. The situation would be become something akin to lawyers being required to do pro bono work. It would mean that the randomly chosen experiment to be duplicated would require a different set of skills than the primary experiment. * Assuming the above, there would be an extremely high impedance in communications between any two of these experiments because no one could really describe their experiment in a way that would allow independence recreation of it. * Smaller institutions would struggle to re-create experiments from better funded institutions. * Getting the second experiment funded would always be difficult because you probably wouldn’t be able to go to the same sources.

naasking 2y ago
> Funding for any experiment would have to include 100% extra because presumably every experiment done would also have to duplicate another, randomly chosen experiment. If this were a universal policy then we'd be no worse off because everyone would face the same challenges. > Smaller institutions would struggle to re-create experiments from better funded institutions. That's already the case. > Getting the second experiment funded would always be difficult because you probably wouldn’t be able to go to the same sources. I thought we were discussing how the original experiment's funding already included funding for the replication?

analog31 2y ago

My experiment was built in a small accelerator lab, plus about $400k in 1990 dollars. Technology was evolving rapidly (lasers, computers), and some of the critical gear was obsolete by the time I finished. Had I needed to secure guaranteed funding for replication, I would not have started the experiment. How it was eventually replicated could not have been known at that point. The thing that comes to mind in this thread is that rules (and rule makers) for small biological and behavioral studies might not make sense for a physics research program, and vice versa.