3 ms·

Your #1 is false. A single unreplicated study is much less meaningful than several successful replications, but it does have some meaning: "this result looks in

by jleader 13y ago

Your #1 is false. A single unreplicated study is much less meaningful than several successful replications, but it does have some meaning: "this result looks interesting, someone should try to replicate it and expand it". If the the first study on any question is completely meaningless, there'd be no motivation to try to replicate it. Regarding #2, it depends what you mean by "making a claim". A study could test a single hypothesis, validate the hypothesis, and then list 10 things that arguably could follow from that hypothesis. For example, after timing the transit of Mercury and validating relativity, a paper could then in its conclusion list other likely consequences of relativity. I think a lot of your statements about "science is ..." are confounding a single study with the entire body of scientific progress. Science requires replication and science requires theoretical explanations. That doesn't mean that a single study always has to provide its own replication (it can't!) or theoretical explanation. An experiment that dis-confirms a theory doesn't have to provide an alternative theory in order to be part of science; the theory may come later.

lutusp 13y ago
> Your #1 is false. A single unreplicated study is much less meaningful than several successful replications, but it does have some meaning ... Not to a scientist. Without replication, a single study is conjecture, interesting but weightless. Are you not aware that an unreplicated study is a standing joke among scientists? Why do you think the gag science journal is named the "Journal of Irreproducibe Results"? Scientists find the name funny because (a) it's an inside joke and (b) it's a truth about science that member of the public just don't get. http://www.jir.com/ http://www.jir.com/ > That doesn't mean that a single study always has to provide its own replication Then it's a good thing I never said that. But a single study has no standing until and unless a disinterested, independent laboratory succeeds in replicating it. This issue is so critical to the future of psychology that, after a series of scandals about replication and credibility, a well-known figure in the field has made an emergency call to create replication circles among psychologists to (a) replicate each other's studies, and thereby try to (b) restore public confidence in the field. He says without a replication program, "I see a train wreck looming." http://www.nature.com/polopoly_fs/7.6716.1349271308!/suppinfoFile/Kahneman%20Letter.pdf http://www.nature.com/polopoly_fs/7.6716.1349271308!/suppinf... (PDF) Quote: "For all these reasons, right or wrong, your field is now the poster child for doubts about the integrity of psychological research. Your problem is not with the few people who have actively challenged the validity of some priming results. It is with the much larger population of colleagues who in the past accepted your surprising results as facts when they were published. ... My reason for writing this letter is that I see a train wreck looming." To see the importance of replication, all you need to do is read the scientific literature.
- jleader 13y ago
  To address your final sentence first, I don't think anyone here has tried to argue against the importance of replication. And yes, I'm familiar with the JIR. I was introduced to it about 35 years ago by my high school's astronomy teacher, Mr. Norm Sperling, now the editor and publisher. I wouldn't take any major actions based on a single study, other than trying to replicate or counter it, but that doesn't mean it has "no meaning". You have legitimate criticisms of the study, but you wrap them in overly-broad generalizations about what is and isn't science. You remind me of an old friend who had such a strict definition of "science" that he claimed that in 4 years as an undergraduate at Caltech, he'd only met one "scientist". So is Prof. Kahneman a scientist?

by jleader 13y ago

lutusp 13y ago

> Your #1 is false. A single unreplicated study is much less meaningful than several successful replications, but it does have some meaning ... Not to a scientist. Without replication, a single study is conjecture, interesting but weightless. Are you not aware that an unreplicated study is a standing joke among scientists? Why do you think the gag science journal is named the "Journal of Irreproducibe Results"? Scientists find the name funny because (a) it's an inside joke and (b) it's a truth about science that member of the public just don't get. http://www.jir.com/ http://www.jir.com/ > That doesn't mean that a single study always has to provide its own replication Then it's a good thing I never said that. But a single study has no standing until and unless a disinterested, independent laboratory succeeds in replicating it. This issue is so critical to the future of psychology that, after a series of scandals about replication and credibility, a well-known figure in the field has made an emergency call to create replication circles among psychologists to (a) replicate each other's studies, and thereby try to (b) restore public confidence in the field. He says without a replication program, "I see a train wreck looming." http://www.nature.com/polopoly_fs/7.6716.1349271308!/suppinfoFile/Kahneman%20Letter.pdf http://www.nature.com/polopoly_fs/7.6716.1349271308!/suppinf... (PDF) Quote: "For all these reasons, right or wrong, your field is now the poster child for doubts about the integrity of psychological research. Your problem is not with the few people who have actively challenged the validity of some priming results. It is with the much larger population of colleagues who in the past accepted your surprising results as facts when they were published. ... My reason for writing this letter is that I see a train wreck looming." To see the importance of replication, all you need to do is read the scientific literature.

jleader 13y ago
To address your final sentence first, I don't think anyone here has tried to argue against the importance of replication. And yes, I'm familiar with the JIR. I was introduced to it about 35 years ago by my high school's astronomy teacher, Mr. Norm Sperling, now the editor and publisher. I wouldn't take any major actions based on a single study, other than trying to replicate or counter it, but that doesn't mean it has "no meaning". You have legitimate criticisms of the study, but you wrap them in overly-broad generalizations about what is and isn't science. You remind me of an old friend who had such a strict definition of "science" that he claimed that in 4 years as an undergraduate at Caltech, he'd only met one "scientist". So is Prof. Kahneman a scientist?