4 ms·

FTA: "“So much in our field is not reproducible, so it’s a huge advantage to understand when data streams might not be reliable,” Schrag says." I am just a lo

FTA: "“So much in our field is not reproducible, so it’s a huge advantage to understand when data streams might not be reliable,” Schrag says." I am just a lowly engineer, but this alarms me. Why is anything that has not been reproduced considered valid science by anyone? Why aren't our standards higher? If you can't reproduce an experimental result, it is useless information is it not? At least an experiment that can be reproduced yet fails to prove a hypothesis can teach you something. An experiment that cannot be reproduced yields no useful information. In fact, it can even mislead! I just don't understand the motivations at play. These are obviously intelligent people who know that you can't fake reality, so why do they publish fraudulent papers? Just for short-term gain? Do they become blinded by belief in their hypothesis?

ncmncm 4y ago
They are blinded by assuming good faith and competence. Most scientists consider most of their colleagues more or less incompetent, and even where they accept experimental results, often reject the experimenter's interpretation of the result, often correctly. Scientists advise us to ignore the abstract, ignore the interpretation, ignore the conclusion, and trust only the data, at most. But we mostly don't get to do that for fields not our own. High prestige is detrimental in that it short-circuits this skepticism. This happens not only in Alzheimer Syndrome work. It put psychology research in the grip of behaviorism, statistics in the grip of non-causality, political science in the grip of dialectics.
niemandhier 4y ago
Biological systems are so complex that reproducing results is extremely hard even if the authors publish their method in detail, simply because they might not know why it works for them. This is hard to understand for software people, since code tends to behave reproducible as a default. Basically the only way to reproduce a difficult finding is to learn the procedure at the original lab. An example: A friend of mine could not reproduce his own findings in another lab. Turned out the precise type of the lamp build into the setup mattered. Another example: I could not reproduce a finding the was something I wanted to build upon. Turned out the precise method to dissolve one of the chemicals in the buffer as the problem. It was even hinted in the paper, but who would describe in detail what he means by “ vigorously stirred” ?
- ncmncm 4y ago
  This is why almost all results from running rats in mazes are spurious. There was an early, very good paper identifying all the details needed to make a valid maze experiment. Nobody cites it, so nobody reads it or acts on its results.
  - summm 4y ago
    Do you happen to have a link to said paper?
    - ncmncm 4y ago
      Sorry, read about it 30 years ago.
- ALittleLight 4y ago
  Biological systems being complicated and unintuitive is an excellent explanation for slow or no progress. It's entirely orthogonal to the question of why published results are not reproducible, misleading, or wrong. If some problem is super hard that explains why I can't answer it, it doesn't explain why I continually publish fake answers. My understanding of the situation is that academics and scientists work in a weird bureaucracy, there is an incentive to publish, academics are very bad at detecting fraud and worse at punishing it and statistical manipulation is easy and endemic. These things explain why there's so much academic research that can't be reproduced and why some academic fields are basically the modern equivalent of astrology.
  - niemandhier 4y ago
    You only notice the cases where you try to reproduce things, and time being limited ,you only reproduce obscure results. As a comparison: I did a bunch of math too and in some cases just accepted the proofs I build on. Same for programming: I neither audited lapac not GCC. For the engineering crowd software dependencies are a probably a good comparison. If they come from a credible source you trust them and build on top of them. Nobody blames companies for not auditing the kernel if there is a security problem in it, but everybody screams: “You should have verified the result”, if some paper is redracted.

ncmncm 4y ago

They are blinded by assuming good faith and competence. Most scientists consider most of their colleagues more or less incompetent, and even where they accept experimental results, often reject the experimenter's interpretation of the result, often correctly. Scientists advise us to ignore the abstract, ignore the interpretation, ignore the conclusion, and trust only the data, at most. But we mostly don't get to do that for fields not our own. High prestige is detrimental in that it short-circuits this skepticism. This happens not only in Alzheimer Syndrome work. It put psychology research in the grip of behaviorism, statistics in the grip of non-causality, political science in the grip of dialectics.

niemandhier 4y ago

Biological systems are so complex that reproducing results is extremely hard even if the authors publish their method in detail, simply because they might not know why it works for them. This is hard to understand for software people, since code tends to behave reproducible as a default. Basically the only way to reproduce a difficult finding is to learn the procedure at the original lab. An example: A friend of mine could not reproduce his own findings in another lab. Turned out the precise type of the lamp build into the setup mattered. Another example: I could not reproduce a finding the was something I wanted to build upon. Turned out the precise method to dissolve one of the chemicals in the buffer as the problem. It was even hinted in the paper, but who would describe in detail what he means by “ vigorously stirred” ?

ncmncm 4y ago
This is why almost all results from running rats in mazes are spurious. There was an early, very good paper identifying all the details needed to make a valid maze experiment. Nobody cites it, so nobody reads it or acts on its results.
- summm 4y ago
  Do you happen to have a link to said paper?
  - ncmncm 4y ago
    Sorry, read about it 30 years ago.
ALittleLight 4y ago
Biological systems being complicated and unintuitive is an excellent explanation for slow or no progress. It's entirely orthogonal to the question of why published results are not reproducible, misleading, or wrong. If some problem is super hard that explains why I can't answer it, it doesn't explain why I continually publish fake answers. My understanding of the situation is that academics and scientists work in a weird bureaucracy, there is an incentive to publish, academics are very bad at detecting fraud and worse at punishing it and statistical manipulation is easy and endemic. These things explain why there's so much academic research that can't be reproduced and why some academic fields are basically the modern equivalent of astrology.
- niemandhier 4y ago
  You only notice the cases where you try to reproduce things, and time being limited ,you only reproduce obscure results. As a comparison: I did a bunch of math too and in some cases just accepted the proofs I build on. Same for programming: I neither audited lapac not GCC. For the engineering crowd software dependencies are a probably a good comparison. If they come from a credible source you trust them and build on top of them. Nobody blames companies for not auditing the kernel if there is a security problem in it, but everybody screams: “You should have verified the result”, if some paper is redracted.