3 ms·

I agree with everything you've said. My experience comes from trying to replicate science that simply isn't reproducible in ways that are not possible to unders

I agree with everything you've said. My experience comes from trying to replicate science that simply isn't reproducible in ways that are not possible to understand from reading a publication. Others I've worked with have similar experiences These disappointing experiences with imperfect science has led me to acknowledge, as you have, that the system is not ideally set up to facilitate good science. In industrial science (pharma companies) it is better as it is harder to hide from bad science but still not perfect I don't fault researchers for not doing perfect science. In many cases funding / resources are not sufficient to allow for exploration of all avenues. Pressure to publish is real. And as you say, independent replication seldom happens as people have incentives to spend time on other things. That is what I mean by a "luxury": if you are fighting for scarce grant dollars, or you are fighting for scarcer tenure, you can't always afford to do work like this. So there is a lot of suboptimal science. It is a sad state of affairs, but I can't blame the researchers for how the system works. If it was as easy as writing robust software tests and having someone else run your code on their computer, there would be little excuse for bad science. But it's harder than that. I see how you can think of that as a cop out, but to me it just seems like the reflection of an imperfect reality

nonbel 9y ago
>"My experience comes from trying to replicate science that simply isn't reproducible in ways that are not possible to understand from reading a publication." Right, this is because no one has done the science. Like Feynman said, someone needs to figure out all the stuff we need to keep track of to be able to do this. I personally saw no interest in doing that (pretty basic stuff like "rat gets more food when giving this drug, therefore the drug made them smarter/healthier"; why not more hungry/motivated, etc?). It was very far from perfection. It was that someone comes up with an assay and mentions some of the limitations at the end of the paper. Then for the next 40 years no one checks this, everyone just continues to run the unverified assay and assume it measures what they want. >"I don't fault researchers for not doing perfect science." I just call it science vs not science, with a hard rule that in science people need to be able to independently replicate your work. Otherwise I call it research (a superset of science), and refer to the people doing it as researchers. I think this is neutral enough, while still allowing the distinction between science vs not. Mainly, if it really is the case, I want it made clear that the current culture/environment/whatever is not allowing a scientific approach to be attempted. Maybe this other method can get somewhere, I dunno. But it does not have a proven history of success like science.

nonbel 9y ago

>"My experience comes from trying to replicate science that simply isn't reproducible in ways that are not possible to understand from reading a publication." Right, this is because no one has done the science. Like Feynman said, someone needs to figure out all the stuff we need to keep track of to be able to do this. I personally saw no interest in doing that (pretty basic stuff like "rat gets more food when giving this drug, therefore the drug made them smarter/healthier"; why not more hungry/motivated, etc?). It was very far from perfection. It was that someone comes up with an assay and mentions some of the limitations at the end of the paper. Then for the next 40 years no one checks this, everyone just continues to run the unverified assay and assume it measures what they want. >"I don't fault researchers for not doing perfect science." I just call it science vs not science, with a hard rule that in science people need to be able to independently replicate your work. Otherwise I call it research (a superset of science), and refer to the people doing it as researchers. I think this is neutral enough, while still allowing the distinction between science vs not. Mainly, if it really is the case, I want it made clear that the current culture/environment/whatever is not allowing a scientific approach to be attempted. Maybe this other method can get somewhere, I dunno. But it does not have a proven history of success like science.