5 ms·

The issue the article doesn't spend too much time on one of the major reasons why reliability is so low: there are too many confounding environmental variables

The issue the article doesn't spend too much time on one of the major reasons why reliability is so low: there are too many confounding environmental variables to control for, so work done in one lab often doesn't generalize One seemingly standard, innocuous piece of equipment can make immune cells "aggressive", while a different brand can make them tolerant. No one knows why. Genetically identical mice ordered from one vendor can consistently have different disease progression compared to mice from another vendor. Background noise in animal facilities can mess with experimental outcomes and even cure disease. Two lots of otherwise identical reagents can act almost like different chemicals Many findings are the result of some set of confounding variables rather than the independent variable. It is impossible to know these things in advance or control for them. The best tactic is to do the same experiment in a different lab with a different research team to control for all these "other factors". Which is expensive (often six of seven figures) and can take months or even years. It is best practice when startups license tech from a university to ask for an exclusivity period where they reproduce the key experiments before committing to a full license Imagine that for every open source library you use, all you have is a detailed read me, a set of tests, and some infrequently commented white board code. You reconstruct the code as best you can (which isn't that hard), but for some reason the tests don't check out. The bug isn't with your software. You call the original researchers and find they used a few libraries that you didn't, and these didn't show up in the publication. You have to manually install all these libraries, test them, and integrate them. Still doesn't work. You realize they used a different os and have to account for that. After six months, still nothing. After visiting their office, you realize the computer used to write the software is on the floor on some very fuzzy carpet, and there is some weird electrostatic bug in the hardware that contributed to how their software performed This is perhaps not the best analogy, and I'm not a scientist myself though have managed scientific teams for years, managed a few tech transfers, and evaluated tons of science as potential investments, but I just wanted to drive home the point of how tough science can be. If others can critique this I would welcome it as it would be great for wet lab scientists and engineers to understand each other's worlds better Of course there is also fraud, cherry picking of data, and careful manipulation of study design to make externalities work in your favor, but a lot of the reliability / reproducibility crisis comes down to lots of uncontrollable / unknowable variables and expensive and long development cycles

nonbel 9y ago
>"Many findings are the result of some set of confounding variables rather than the independent variable. It is impossible to know these things in advance or control for them." It is literally the primary job of the (experimentalist) scientist to figure this stuff out. You are describing people skipping the science and trying to jump directly to the conclusions. This is exactly what Richard Feynman called "cargo cult science": "For example, there have been many experiments running rats through all kinds of mazes, and so on—with little clear result. But in 1937 a man named Young did a very interesting one. He had a long corridor with doors all along one side where the rats came in, and doors along the other side where the food was. He wanted to see if he could train the rats to go in at the third door down from wherever he started them off. No. The rats went immediately to the door where the food had been the time before. The question was, how did the rats know, because the corridor was so beautifully built and so uniform, that this was the same door as before? Obviously there was something about the door that was different from the other doors. So he painted the doors very carefully, arranging the textures on the faces of the doors exactly the same. Still the rats could tell. Then he thought maybe the rats were smelling the food, so he used chemicals to change the smell after each run. Still the rats could tell. Then he realized the rats might be able to tell by seeing the lights and the arrangement in the laboratory like any commonsense person. So he covered the corridor, and, still the rats could tell. He finally found that they could tell by the way the floor sounded when they ran over it. And he could only fix that by putting his corridor in sand. So he covered one after another of all possible clues and finally was able to fool the rats so that they had to learn to go in the third door. If he relaxed any of his conditions, the rats could tell. Now, from a scientific standpoint, that is an A‑Number‑l experiment. That is the experiment that makes rat‑running experiments sensible, because it uncovers the clues that the rat is really using—not what you think it’s using. And that is the experiment that tells exactly what conditions you have to use in order to be careful and control everything in an experiment with rat‑running. I looked into the subsequent history of this research. The subsequent experiment, and the one after that, never referred to Mr. Young. They never used any of his criteria of putting the corridor on sand, or being very careful. They just went right on running rats in the same old way, and paid no attention to the great discoveries of Mr. Young, and his papers are not referred to, because he didn’t discover anything about the rats. In fact, he discovered all the things you have to do to discover something about rats. But not paying attention to experiments like that is a characteristic of Cargo Cult Science." http://calteches.library.caltech.edu/51/2/CargoCult.htm http://calteches.library.caltech.edu/51/2/CargoCult.htm

nonbel 9y ago

>"Many findings are the result of some set of confounding variables rather than the independent variable. It is impossible to know these things in advance or control for them." It is literally the primary job of the (experimentalist) scientist to figure this stuff out. You are describing people skipping the science and trying to jump directly to the conclusions. This is exactly what Richard Feynman called "cargo cult science": "For example, there have been many experiments running rats through all kinds of mazes, and so on—with little clear result. But in 1937 a man named Young did a very interesting one. He had a long corridor with doors all along one side where the rats came in, and doors along the other side where the food was. He wanted to see if he could train the rats to go in at the third door down from wherever he started them off. No. The rats went immediately to the door where the food had been the time before. The question was, how did the rats know, because the corridor was so beautifully built and so uniform, that this was the same door as before? Obviously there was something about the door that was different from the other doors. So he painted the doors very carefully, arranging the textures on the faces of the doors exactly the same. Still the rats could tell. Then he thought maybe the rats were smelling the food, so he used chemicals to change the smell after each run. Still the rats could tell. Then he realized the rats might be able to tell by seeing the lights and the arrangement in the laboratory like any commonsense person. So he covered the corridor, and, still the rats could tell. He finally found that they could tell by the way the floor sounded when they ran over it. And he could only fix that by putting his corridor in sand. So he covered one after another of all possible clues and finally was able to fool the rats so that they had to learn to go in the third door. If he relaxed any of his conditions, the rats could tell. Now, from a scientific standpoint, that is an A‑Number‑l experiment. That is the experiment that makes rat‑running experiments sensible, because it uncovers the clues that the rat is really using—not what you think it’s using. And that is the experiment that tells exactly what conditions you have to use in order to be careful and control everything in an experiment with rat‑running. I looked into the subsequent history of this research. The subsequent experiment, and the one after that, never referred to Mr. Young. They never used any of his criteria of putting the corridor on sand, or being very careful. They just went right on running rats in the same old way, and paid no attention to the great discoveries of Mr. Young, and his papers are not referred to, because he didn’t discover anything about the rats. In fact, he discovered all the things you have to do to discover something about rats. But not paying attention to experiments like that is a characteristic of Cargo Cult Science." http://calteches.library.caltech.edu/51/2/CargoCult.htm http://calteches.library.caltech.edu/51/2/CargoCult.htm