6 ms·

Richard Feynman had a speech warning about this problem back in 1974. Complete with a story about how a student was excited about doing some replication experim

Richard Feynman had a speech warning about this problem back in 1974. Complete with a story about how a student was excited about doing some replication experiments on rats and was told "no, you cannot do that, because the experiment has already been done and you would be wasting time". Others have also been worried about replication problems from the 1960's on. Hopefully some of this worry sticks this time and we can get a better understanding of what is really true in these fields. Physics likes to have very well defined uncertainties on everything they observe and don't like to say something is "true" unless the fact is likely at 5 to 7 sigma. That seems like a good margin. In psychology, 2 sigma is the standard for publishing a result. [1]http://calteches.library.caltech.edu/51/2/CargoCult.htm http://calteches.library.caltech.edu/51/2/CargoCult.htm

cultus 8y ago
P-values are not good measures for reliability, because they absolutely are not the same thing as the chance that a result is wrong. This confusion, and ignorance of the prior probability of hypotheses is the reason why replication rates are so bad. In physics you can get away with not caring about the distinction, because of how accurate the measurements and precise the theories are. That doesn't fly in most other sciences, though.
- jmcgough 8y ago
  > P-values are not good measures for reliability, because they absolutely are not the same thing as the chance that a result is wrong As an undergrad, I even had tenured professors try to tell me that a p-value is the chance that a result is wrong. Most researchers in psych or bio sciences have a weak understanding of statistics, usually taking a single statistics class in undergrad, then a single one in grad.