3 ms·

>"Many findings are the result of some set of confounding variables rather than the independent variable. It is impossible to know these things in advance or co

by nonbel 9y ago

>"Many findings are the result of some set of confounding variables rather than the independent variable. It is impossible to know these things in advance or control for them." It is literally the primary job of the (experimentalist) scientist to figure this stuff out. You are describing people skipping the science and trying to jump directly to the conclusions. This is exactly what Richard Feynman called "cargo cult science": "For example, there have been many experiments running rats through all kinds of mazes, and so on—with little clear result. But in 1937 a man named Young did a very interesting one. He had a long corridor with doors all along one side where the rats came in, and doors along the other side where the food was. He wanted to see if he could train the rats to go in at the third door down from wherever he started them off. No. The rats went immediately to the door where the food had been the time before. The question was, how did the rats know, because the corridor was so beautifully built and so uniform, that this was the same door as before? Obviously there was something about the door that was different from the other doors. So he painted the doors very carefully, arranging the textures on the faces of the doors exactly the same. Still the rats could tell. Then he thought maybe the rats were smelling the food, so he used chemicals to change the smell after each run. Still the rats could tell. Then he realized the rats might be able to tell by seeing the lights and the arrangement in the laboratory like any commonsense person. So he covered the corridor, and, still the rats could tell. He finally found that they could tell by the way the floor sounded when they ran over it. And he could only fix that by putting his corridor in sand. So he covered one after another of all possible clues and finally was able to fool the rats so that they had to learn to go in the third door. If he relaxed any of his conditions, the rats could tell. Now, from a scientific standpoint, that is an A‑Number‑l experiment. That is the experiment that makes rat‑running experiments sensible, because it uncovers the clues that the rat is really using—not what you think it’s using. And that is the experiment that tells exactly what conditions you have to use in order to be careful and control everything in an experiment with rat‑running. I looked into the subsequent history of this research. The subsequent experiment, and the one after that, never referred to Mr. Young. They never used any of his criteria of putting the corridor on sand, or being very careful. They just went right on running rats in the same old way, and paid no attention to the great discoveries of Mr. Young, and his papers are not referred to, because he didn’t discover anything about the rats. In fact, he discovered all the things you have to do to discover something about rats. But not paying attention to experiments like that is a characteristic of Cargo Cult Science." http://calteches.library.caltech.edu/51/2/CargoCult.htm http://calteches.library.caltech.edu/51/2/CargoCult.htm

aaavl2821 9y ago
Do you think that many experimentalists actually are able to / have the luxury of figuring all this stuff out if an experiment works? Of course when stuff goes wrong you investigate all confounding factors until you make it work, but that doesn't mean you've chased down every loose end. And if you are doing industrial drug discovery and development, then validation, documentation and quality are absolute priorities. But from my experience (again, managing tech transfers, not being an experimentalist), people solve for getting something to work with some level of reliability, and you simply cannot test all confounding factors To use the Feynman example: experimentalists do all of the things the rat researcher did to "debug" an experiment. But you can do the same study in a different lab, and there may be another environmental variable you have to control for that didnt exist in the first lab. Especially when you have experiments that are orders of magnitude more complex than the rat experiment My perspective is that in theory you can do perfect science and figure out all the variables but in reality you just can't. Again, not a scientist, but I think the reproducibility issues in research and low success rates in drug development support that perspective If you are a mathematician, there is basically no expected failure rate. You prove it or you don't. If you are an engineer, some stuff just isn't possible, and unpredictable stuff happens, but many times there's a right answer. If you are a psychologist, if you get the treatment right half the time you are good. If you are a drug researcher, even getting one approved drugs is phenomenal. It's by nature a field with low technical success rates
- nonbel 9y ago
  >"Do you think that many experimentalists actually are able to / have the luxury of figuring all this stuff out if an experiment works?" It is a crucial part of science, not a luxury. Failure to do so means you are doing something else (I use the general term "research"). >"But you can do the same study in a different lab, and there may be another environmental variable you have to control for that didnt exist in the first lab." Yes, independent replication is required to be sure you understand the experimental conditions. >"My perspective is that in theory you can do perfect science and figure out all the variables but in reality you just can't... It's by nature a field with low technical success rates" Perhaps, or perhaps this is an excuse by people who haven't even tried to actually approach the problem scientifically. They (and/or the funding agencies) just want to jump right to the exciting discoveries and cures instead of doing the science. Since the scientific approach has not been tried, we don't know. >"the reproducibility issues in research and low success rates in drug development support that perspective" They also support the perspective that sloppy research wastes insane amounts of time and money. Jumping to the "it is so complicated" excuse without actually trying to do things correctly seems really disingenuous. Anyway, I left biomed exactly for this reason. There did not seem to be any interest in doing actual science. No one was ever going to try a direct replication of my work, no one cared about developing and testing quantitative models (only coming up with meaningless "significant" p-values), etc.

by nonbel 9y ago

aaavl2821 9y ago

Do you think that many experimentalists actually are able to / have the luxury of figuring all this stuff out if an experiment works? Of course when stuff goes wrong you investigate all confounding factors until you make it work, but that doesn't mean you've chased down every loose end. And if you are doing industrial drug discovery and development, then validation, documentation and quality are absolute priorities. But from my experience (again, managing tech transfers, not being an experimentalist), people solve for getting something to work with some level of reliability, and you simply cannot test all confounding factors To use the Feynman example: experimentalists do all of the things the rat researcher did to "debug" an experiment. But you can do the same study in a different lab, and there may be another environmental variable you have to control for that didnt exist in the first lab. Especially when you have experiments that are orders of magnitude more complex than the rat experiment My perspective is that in theory you can do perfect science and figure out all the variables but in reality you just can't. Again, not a scientist, but I think the reproducibility issues in research and low success rates in drug development support that perspective If you are a mathematician, there is basically no expected failure rate. You prove it or you don't. If you are an engineer, some stuff just isn't possible, and unpredictable stuff happens, but many times there's a right answer. If you are a psychologist, if you get the treatment right half the time you are good. If you are a drug researcher, even getting one approved drugs is phenomenal. It's by nature a field with low technical success rates

nonbel 9y ago
>"Do you think that many experimentalists actually are able to / have the luxury of figuring all this stuff out if an experiment works?" It is a crucial part of science, not a luxury. Failure to do so means you are doing something else (I use the general term "research"). >"But you can do the same study in a different lab, and there may be another environmental variable you have to control for that didnt exist in the first lab." Yes, independent replication is required to be sure you understand the experimental conditions. >"My perspective is that in theory you can do perfect science and figure out all the variables but in reality you just can't... It's by nature a field with low technical success rates" Perhaps, or perhaps this is an excuse by people who haven't even tried to actually approach the problem scientifically. They (and/or the funding agencies) just want to jump right to the exciting discoveries and cures instead of doing the science. Since the scientific approach has not been tried, we don't know. >"the reproducibility issues in research and low success rates in drug development support that perspective" They also support the perspective that sloppy research wastes insane amounts of time and money. Jumping to the "it is so complicated" excuse without actually trying to do things correctly seems really disingenuous. Anyway, I left biomed exactly for this reason. There did not seem to be any interest in doing actual science. No one was ever going to try a direct replication of my work, no one cared about developing and testing quantitative models (only coming up with meaningless "significant" p-values), etc.