4 ms·

Do you think that many experimentalists actually are able to / have the luxury of figuring all this stuff out if an experiment works? Of course when stuff goes

Do you think that many experimentalists actually are able to / have the luxury of figuring all this stuff out if an experiment works? Of course when stuff goes wrong you investigate all confounding factors until you make it work, but that doesn't mean you've chased down every loose end. And if you are doing industrial drug discovery and development, then validation, documentation and quality are absolute priorities. But from my experience (again, managing tech transfers, not being an experimentalist), people solve for getting something to work with some level of reliability, and you simply cannot test all confounding factors To use the Feynman example: experimentalists do all of the things the rat researcher did to "debug" an experiment. But you can do the same study in a different lab, and there may be another environmental variable you have to control for that didnt exist in the first lab. Especially when you have experiments that are orders of magnitude more complex than the rat experiment My perspective is that in theory you can do perfect science and figure out all the variables but in reality you just can't. Again, not a scientist, but I think the reproducibility issues in research and low success rates in drug development support that perspective If you are a mathematician, there is basically no expected failure rate. You prove it or you don't. If you are an engineer, some stuff just isn't possible, and unpredictable stuff happens, but many times there's a right answer. If you are a psychologist, if you get the treatment right half the time you are good. If you are a drug researcher, even getting one approved drugs is phenomenal. It's by nature a field with low technical success rates

nonbel 9y ago
>"Do you think that many experimentalists actually are able to / have the luxury of figuring all this stuff out if an experiment works?" It is a crucial part of science, not a luxury. Failure to do so means you are doing something else (I use the general term "research"). >"But you can do the same study in a different lab, and there may be another environmental variable you have to control for that didnt exist in the first lab." Yes, independent replication is required to be sure you understand the experimental conditions. >"My perspective is that in theory you can do perfect science and figure out all the variables but in reality you just can't... It's by nature a field with low technical success rates" Perhaps, or perhaps this is an excuse by people who haven't even tried to actually approach the problem scientifically. They (and/or the funding agencies) just want to jump right to the exciting discoveries and cures instead of doing the science. Since the scientific approach has not been tried, we don't know. >"the reproducibility issues in research and low success rates in drug development support that perspective" They also support the perspective that sloppy research wastes insane amounts of time and money. Jumping to the "it is so complicated" excuse without actually trying to do things correctly seems really disingenuous. Anyway, I left biomed exactly for this reason. There did not seem to be any interest in doing actual science. No one was ever going to try a direct replication of my work, no one cared about developing and testing quantitative models (only coming up with meaningless "significant" p-values), etc.
- aaavl2821 9y ago
  I agree with everything you've said. My experience comes from trying to replicate science that simply isn't reproducible in ways that are not possible to understand from reading a publication. Others I've worked with have similar experiences These disappointing experiences with imperfect science has led me to acknowledge, as you have, that the system is not ideally set up to facilitate good science. In industrial science (pharma companies) it is better as it is harder to hide from bad science but still not perfect I don't fault researchers for not doing perfect science. In many cases funding / resources are not sufficient to allow for exploration of all avenues. Pressure to publish is real. And as you say, independent replication seldom happens as people have incentives to spend time on other things. That is what I mean by a "luxury": if you are fighting for scarce grant dollars, or you are fighting for scarcer tenure, you can't always afford to do work like this. So there is a lot of suboptimal science. It is a sad state of affairs, but I can't blame the researchers for how the system works. If it was as easy as writing robust software tests and having someone else run your code on their computer, there would be little excuse for bad science. But it's harder than that. I see how you can think of that as a cop out, but to me it just seems like the reflection of an imperfect reality
  - nonbel 9y ago
    >"My experience comes from trying to replicate science that simply isn't reproducible in ways that are not possible to understand from reading a publication." Right, this is because no one has done the science. Like Feynman said, someone needs to figure out all the stuff we need to keep track of to be able to do this. I personally saw no interest in doing that (pretty basic stuff like "rat gets more food when giving this drug, therefore the drug made them smarter/healthier"; why not more hungry/motivated, etc?). It was very far from perfection. It was that someone comes up with an assay and mentions some of the limitations at the end of the paper. Then for the next 40 years no one checks this, everyone just continues to run the unverified assay and assume it measures what they want. >"I don't fault researchers for not doing perfect science." I just call it science vs not science, with a hard rule that in science people need to be able to independently replicate your work. Otherwise I call it research (a superset of science), and refer to the people doing it as researchers. I think this is neutral enough, while still allowing the distinction between science vs not. Mainly, if it really is the case, I want it made clear that the current culture/environment/whatever is not allowing a scientific approach to be attempted. Maybe this other method can get somewhere, I dunno. But it does not have a proven history of success like science.

nonbel 9y ago

>"Do you think that many experimentalists actually are able to / have the luxury of figuring all this stuff out if an experiment works?" It is a crucial part of science, not a luxury. Failure to do so means you are doing something else (I use the general term "research"). >"But you can do the same study in a different lab, and there may be another environmental variable you have to control for that didnt exist in the first lab." Yes, independent replication is required to be sure you understand the experimental conditions. >"My perspective is that in theory you can do perfect science and figure out all the variables but in reality you just can't... It's by nature a field with low technical success rates" Perhaps, or perhaps this is an excuse by people who haven't even tried to actually approach the problem scientifically. They (and/or the funding agencies) just want to jump right to the exciting discoveries and cures instead of doing the science. Since the scientific approach has not been tried, we don't know. >"the reproducibility issues in research and low success rates in drug development support that perspective" They also support the perspective that sloppy research wastes insane amounts of time and money. Jumping to the "it is so complicated" excuse without actually trying to do things correctly seems really disingenuous. Anyway, I left biomed exactly for this reason. There did not seem to be any interest in doing actual science. No one was ever going to try a direct replication of my work, no one cared about developing and testing quantitative models (only coming up with meaningless "significant" p-values), etc.

aaavl2821 9y ago
I agree with everything you've said. My experience comes from trying to replicate science that simply isn't reproducible in ways that are not possible to understand from reading a publication. Others I've worked with have similar experiences These disappointing experiences with imperfect science has led me to acknowledge, as you have, that the system is not ideally set up to facilitate good science. In industrial science (pharma companies) it is better as it is harder to hide from bad science but still not perfect I don't fault researchers for not doing perfect science. In many cases funding / resources are not sufficient to allow for exploration of all avenues. Pressure to publish is real. And as you say, independent replication seldom happens as people have incentives to spend time on other things. That is what I mean by a "luxury": if you are fighting for scarce grant dollars, or you are fighting for scarcer tenure, you can't always afford to do work like this. So there is a lot of suboptimal science. It is a sad state of affairs, but I can't blame the researchers for how the system works. If it was as easy as writing robust software tests and having someone else run your code on their computer, there would be little excuse for bad science. But it's harder than that. I see how you can think of that as a cop out, but to me it just seems like the reflection of an imperfect reality
- nonbel 9y ago
  >"My experience comes from trying to replicate science that simply isn't reproducible in ways that are not possible to understand from reading a publication." Right, this is because no one has done the science. Like Feynman said, someone needs to figure out all the stuff we need to keep track of to be able to do this. I personally saw no interest in doing that (pretty basic stuff like "rat gets more food when giving this drug, therefore the drug made them smarter/healthier"; why not more hungry/motivated, etc?). It was very far from perfection. It was that someone comes up with an assay and mentions some of the limitations at the end of the paper. Then for the next 40 years no one checks this, everyone just continues to run the unverified assay and assume it measures what they want. >"I don't fault researchers for not doing perfect science." I just call it science vs not science, with a hard rule that in science people need to be able to independently replicate your work. Otherwise I call it research (a superset of science), and refer to the people doing it as researchers. I think this is neutral enough, while still allowing the distinction between science vs not. Mainly, if it really is the case, I want it made clear that the current culture/environment/whatever is not allowing a scientific approach to be attempted. Maybe this other method can get somewhere, I dunno. But it does not have a proven history of success like science.