4 ms·

The article argues the replication crisis is somehow unique to psychology, but it's not. As the Scott Alexander essay makes clear, it also affects psychiatry a

The article argues the replication crisis is somehow unique to psychology, but it's not. As the Scott Alexander essay makes clear, it also affects psychiatry and it's apparently the case that many areas of medicine have this problem. Biotech studies are also hives of replication failures. Even AI research has had replication failures and that's based on running theoretically deterministic software on theoretically deterministic machines! Some of this is that accurately describing studies and replicating them is hard. But some of it is that academics aren't incentivised to find the truth, but rather, to make it appear that academics know a lot of things.

rleigh 7y ago
I'm unsurprised by the replication crisis. I've seen scientists p-hacking, and even choosing inappropriate statistical tests because it gave a "better" result. Example: using a non-parametric "U"-test when a parametric "T"-test gave a non-significant result. Along with low numbers of replicates. Statistical analysis is only useful and meaningful if your data is of good quality, and you use a statistical test appropriate for the data. If your data is of marginal quality, then I'm afraid that it's simply unworthy of publication. But when your career hangs in the balance, stuff like this gets through. The major problem with current scientific practice is that good practice is actively penalised. I used to work in industry, and I was shocked at the lax standard of work, particularly with respect to accuracy and precision, of wet lab scientists in university research settings. I mentioned this to a few postdocs over the years and paraphrased was told that "if it's good enough to publish, then it's good enough for me", which if you think about it, is actually quite a low bar. Most of the people were fully aware they were doing sloppy work, but didn't care. How can it be improved? I think there are two sides to this coin. Firstly, good practice has to be encouraged and rewarded, and sloppiness penalised. That requires a culture change in the laboratories. Too many PIs don't care about what happens in their labs so long as "good" results are being generated by their underlings. They don't look after instrument calibration and ensure that people are working to GLP standards. In industrial labs, we had to send samples off to reference labs, analyse random samples provided to us, and undergo inspections and audits to prove we were providing correct analyses. Maybe academic labs should be obligated to prove themselves as well, or lose their funding? Secondly, a project delivering negative results should not be a career-ending move. Failing experiments does not necessarily mean one is a bad scientist. But right now, the incentives are to spin all results in a positive light, even if it means publishing bad science, because that's what it takes to keep the funding coming in. Success should be rewarded, but I think our criteria for what success is need to be recalibrated to reduce charlatans abusing the system for their own benefit. Publishing a paper isn't enough; it's got to be replicable independently.
- repolfx 7y ago
  I think you're right, and your experiences of corporate vs academic research quality matches my own, more or less (in different fields). But this does lead to the question of - why not just reduce academic funding, matched by corresponding cuts in corporation taxes? That would obviously not lead to a 1:1 transfer of research funding or anything even close to it, but if the replication crisis seems to suggest anything at all it's that there's too many scientists chasing too little real knowledge, with too few reality checks of the sort industrial labs require. If more funding was from industry, the quality of science might be higher.

rleigh 7y ago

I'm unsurprised by the replication crisis. I've seen scientists p-hacking, and even choosing inappropriate statistical tests because it gave a "better" result. Example: using a non-parametric "U"-test when a parametric "T"-test gave a non-significant result. Along with low numbers of replicates. Statistical analysis is only useful and meaningful if your data is of good quality, and you use a statistical test appropriate for the data. If your data is of marginal quality, then I'm afraid that it's simply unworthy of publication. But when your career hangs in the balance, stuff like this gets through. The major problem with current scientific practice is that good practice is actively penalised. I used to work in industry, and I was shocked at the lax standard of work, particularly with respect to accuracy and precision, of wet lab scientists in university research settings. I mentioned this to a few postdocs over the years and paraphrased was told that "if it's good enough to publish, then it's good enough for me", which if you think about it, is actually quite a low bar. Most of the people were fully aware they were doing sloppy work, but didn't care. How can it be improved? I think there are two sides to this coin. Firstly, good practice has to be encouraged and rewarded, and sloppiness penalised. That requires a culture change in the laboratories. Too many PIs don't care about what happens in their labs so long as "good" results are being generated by their underlings. They don't look after instrument calibration and ensure that people are working to GLP standards. In industrial labs, we had to send samples off to reference labs, analyse random samples provided to us, and undergo inspections and audits to prove we were providing correct analyses. Maybe academic labs should be obligated to prove themselves as well, or lose their funding? Secondly, a project delivering negative results should not be a career-ending move. Failing experiments does not necessarily mean one is a bad scientist. But right now, the incentives are to spin all results in a positive light, even if it means publishing bad science, because that's what it takes to keep the funding coming in. Success should be rewarded, but I think our criteria for what success is need to be recalibrated to reduce charlatans abusing the system for their own benefit. Publishing a paper isn't enough; it's got to be replicable independently.

repolfx 7y ago
I think you're right, and your experiences of corporate vs academic research quality matches my own, more or less (in different fields). But this does lead to the question of - why not just reduce academic funding, matched by corresponding cuts in corporation taxes? That would obviously not lead to a 1:1 transfer of research funding or anything even close to it, but if the replication crisis seems to suggest anything at all it's that there's too many scientists chasing too little real knowledge, with too few reality checks of the sort industrial labs require. If more funding was from industry, the quality of science might be higher.