3 ms·

This doesn’t scale. Sometimes only a small number of labs have the equipment needed to run an experiment, let alone the training on the equipment to do what nee

This doesn’t scale. Sometimes only a small number of labs have the equipment needed to run an experiment, let alone the training on the equipment to do what needs to be done. This equipment can cost millions. Even if the results are correct, experimental science is very finicky by its very nature. Getting an experiment to work can take months.

Voultapher 3y ago
I'd say if no one can afford to, or knows how to replicate the thing you did. What point is there in it? And only because such an approach has downsides it's not clear it means the overall benefit of lessening the replication crisis and forcing scientists to provide enough information their work can be replicated at all would be a huge win. Based on people like https://www.youtube.com/@AppliedScience https://www.youtube.com/@AppliedScience experience, 9 out of 10 times there is some crucial information missing in the papers they are trying to replicate.
- anonymouskimmer 3y ago
  > I'd say if no one can afford to Because sometimes they can afford to build on your prior results, and advance the state of the art. Factoring in all of the malfeasance, what's the trade off between not publishing due to inability/unwillingness to replicate, and publishing bad results? We should know this tradeoff before significantly changing the status quo. > or knows how to replicate the thing you did. Agree completely.
somenameforme 3y ago
Like what? I can think of extremely few domains that any decent research university isn't well equipped to competently replicate. One of the very few benefits of the tuition explosion - even undergrads get relatively casual access to equipment worth millions of dollars. Of course they can't replicate things like ultra high energy particle research, but these sort of obscure things make up a very negligible chunk of all science produced, even if it's quite an important little chunk.
- cycomanic 3y ago
  I guess you're not thinking hard enough. I estimate >90% of research in most technical fields would require 10s of years to replicate unless you are one of the few labs already working on the same topic. For example in on of the research areas I'm familiar with (optical communications), there are maybe 10 academic labs in Europe (and even less in the US) who have the equipment to reproduce some of our experiments. In our lab there is 1 PhD student who could pull off reproducing the more sophisticated experiments (because he is the one focusing on communications) it took him 2 years to get to that stage. This is an relatively easy area, i.e. equipment is largely off the shelve, very applied with lots of industry involvement. There are plenty of experiments published which could only be done in 2 labs (both of them industrial), just due to the cost of the required equipment. In other areas (e.g. with fabrication in the clean room) reproduction would require even more time investment. Don't get me wrong, reproducing results is important, but what people don't realise it happens all the time when people do adopt part of published results into their research. Mandatory reproducing results would just create large overheads which would get us nowhere.
  - somenameforme 3y ago
    It's entirely possible there are fields with needs I am unaware of or not considering, but your response is not compelling and sounds like hand-waiving. Exactly what equipment are you talking about? I suspect you are likely grossly underestimating the available supplies at many research university in the US. For instance things like class 100 clean rooms are basic facilities. Many (and I want to say most) research universities also have partnerships with (if not ownership) of various specialized labs in the surrounding areas for more specific purposes. For instance the NASA Jet Propulsion Lab is managed by Caltech.
    - cycomanic 3y ago
      Regarding equipment: Realtime oscilloscope at least 4 channels > 50 GHz bandwidth $0.5M (for some research you need >=12 channels so multiply that number) Arbitrary waveform generator 4 channels > 45 GHz bandwidth $0.3M (again you might need more than for channels) RF amplifiers, electro-optic components etc. easily cost $2000-$5000 each and you need several (4-8 at least) of these. The RF cables and connectors/adapters easily cost several thousand $ each. Fibre components, subsystem components (e.g. a WSS of which you will likely need 4 or so is $50k). And I will certainly not let a student without training touch the sensitive high-speed RF equipment. Regarding your comment on clean-room. For many fabrication purposes class 100 is not sufficient (also calling it basic facility is quite rich). And the equipment in is very expensive, LPCVD machines, E-beam, other lithography ... is $10sM. Most universities I'm aware of require fees (typically paid from grants) of $10sk per year to use the facilities (those are the reduced rates for university staff). The training/certification on the equipment typically takes about 1year. Regarding JPL, yes it's managed by Caltech, what do you think NASA will say if Caltech professors will ask for a student to use the facilities to verify some paper? Sure, lets delay the next Mars mission a year or so, to let some PhD students try stuff in the labs. I think you seriously underestimate what the cost of using all that equipment is and how much training is involved to be allowed to use it. You definitely don't want any
rtsil 3y ago
Fine, let's just do it on those that scale. Not all science is done in high-end, expensive labs, and this type of control will help the young researchers who are publishing their first works to gain more credibility and develop healthy habits. Sadly, as long as the tyranny of publishing exists, the other researchers will always prefer working on their own things than replicating someone else's not yet published experiment.
etrautmann 3y ago
Agreed - having done a PhD in an experimental neuroscience lab, there's a nonsignificant number of things that nobody else in the world, even in my own lab, can do. I can train the techniques to others, and do, but this is separate from scientific discovery. There are no incentives for someone else to spend their time replicating my work using unbelievably challenging and expensive methods. It just wouldn't work in practice without a more fundamental restructuring of the whole enterprise (which is also necessary but hard).

Voultapher 3y ago

I'd say if no one can afford to, or knows how to replicate the thing you did. What point is there in it? And only because such an approach has downsides it's not clear it means the overall benefit of lessening the replication crisis and forcing scientists to provide enough information their work can be replicated at all would be a huge win. Based on people like https://www.youtube.com/@AppliedScience https://www.youtube.com/@AppliedScience experience, 9 out of 10 times there is some crucial information missing in the papers they are trying to replicate.

anonymouskimmer 3y ago
> I'd say if no one can afford to Because sometimes they can afford to build on your prior results, and advance the state of the art. Factoring in all of the malfeasance, what's the trade off between not publishing due to inability/unwillingness to replicate, and publishing bad results? We should know this tradeoff before significantly changing the status quo. > or knows how to replicate the thing you did. Agree completely.

somenameforme 3y ago

Like what? I can think of extremely few domains that any decent research university isn't well equipped to competently replicate. One of the very few benefits of the tuition explosion - even undergrads get relatively casual access to equipment worth millions of dollars. Of course they can't replicate things like ultra high energy particle research, but these sort of obscure things make up a very negligible chunk of all science produced, even if it's quite an important little chunk.

cycomanic 3y ago
I guess you're not thinking hard enough. I estimate >90% of research in most technical fields would require 10s of years to replicate unless you are one of the few labs already working on the same topic. For example in on of the research areas I'm familiar with (optical communications), there are maybe 10 academic labs in Europe (and even less in the US) who have the equipment to reproduce some of our experiments. In our lab there is 1 PhD student who could pull off reproducing the more sophisticated experiments (because he is the one focusing on communications) it took him 2 years to get to that stage. This is an relatively easy area, i.e. equipment is largely off the shelve, very applied with lots of industry involvement. There are plenty of experiments published which could only be done in 2 labs (both of them industrial), just due to the cost of the required equipment. In other areas (e.g. with fabrication in the clean room) reproduction would require even more time investment. Don't get me wrong, reproducing results is important, but what people don't realise it happens all the time when people do adopt part of published results into their research. Mandatory reproducing results would just create large overheads which would get us nowhere.
- somenameforme 3y ago
  It's entirely possible there are fields with needs I am unaware of or not considering, but your response is not compelling and sounds like hand-waiving. Exactly what equipment are you talking about? I suspect you are likely grossly underestimating the available supplies at many research university in the US. For instance things like class 100 clean rooms are basic facilities. Many (and I want to say most) research universities also have partnerships with (if not ownership) of various specialized labs in the surrounding areas for more specific purposes. For instance the NASA Jet Propulsion Lab is managed by Caltech.
  - cycomanic 3y ago
    Regarding equipment: Realtime oscilloscope at least 4 channels > 50 GHz bandwidth $0.5M (for some research you need >=12 channels so multiply that number) Arbitrary waveform generator 4 channels > 45 GHz bandwidth $0.3M (again you might need more than for channels) RF amplifiers, electro-optic components etc. easily cost $2000-$5000 each and you need several (4-8 at least) of these. The RF cables and connectors/adapters easily cost several thousand $ each. Fibre components, subsystem components (e.g. a WSS of which you will likely need 4 or so is $50k). And I will certainly not let a student without training touch the sensitive high-speed RF equipment. Regarding your comment on clean-room. For many fabrication purposes class 100 is not sufficient (also calling it basic facility is quite rich). And the equipment in is very expensive, LPCVD machines, E-beam, other lithography ... is $10sM. Most universities I'm aware of require fees (typically paid from grants) of $10sk per year to use the facilities (those are the reduced rates for university staff). The training/certification on the equipment typically takes about 1year. Regarding JPL, yes it's managed by Caltech, what do you think NASA will say if Caltech professors will ask for a student to use the facilities to verify some paper? Sure, lets delay the next Mars mission a year or so, to let some PhD students try stuff in the labs. I think you seriously underestimate what the cost of using all that equipment is and how much training is involved to be allowed to use it. You definitely don't want any

rtsil 3y ago

Fine, let's just do it on those that scale. Not all science is done in high-end, expensive labs, and this type of control will help the young researchers who are publishing their first works to gain more credibility and develop healthy habits. Sadly, as long as the tyranny of publishing exists, the other researchers will always prefer working on their own things than replicating someone else's not yet published experiment.

etrautmann 3y ago

Agreed - having done a PhD in an experimental neuroscience lab, there's a nonsignificant number of things that nobody else in the world, even in my own lab, can do. I can train the techniques to others, and do, but this is separate from scientific discovery. There are no incentives for someone else to spend their time replicating my work using unbelievably challenging and expensive methods. It just wouldn't work in practice without a more fundamental restructuring of the whole enterprise (which is also necessary but hard).