10 ms·

Some obvious responses occur to a skeptical reader of science: 1. This study has precisely no meaning until it has been replicated, preferably with a larger se

by lutusp 14y ago

Some obvious responses occur to a skeptical reader of science: 1. This study has precisely no meaning until it has been replicated, preferably with a larger set of experimental subjects. 2. Science is not one study making ten claims, it is ten studies making one claim. This study falls into the former category. 3. The study describes, it doesn't try to explain. Science requires a testable explanation, one that can be generalized and potentially falsified by independent laboratories. If the study had offered a possible, testable explanation, it would have crossed the threshold of science. 4. It's important to say that psychological studies are virtually never replicated. One reason is that the original studies tend to maker nebulous claims that are difficult to quantify (like this study does). Another is that psychological studies tend not to be accompanied by the original study data, to a greater extent than studies in scientific disciplines. A third reason is that psychological journals tend to reject replication papers, especially those that don't confirm the original study's findings.

carbocation 14y ago
I was going to write in disagreement with your point #3, but then I went back and re-read the Atlantic article. In that context, I can see why you would write that. However, I think it needs this context, so to others who were going to knee-jerk react to point #3, re-read the article first. The study is not impressive. Most studies aren't, and exploratory, early-stage studies rarely can be. It's OK that early studies of a phenomenon are not impressive, but the tests do seem to be extremely narrow. The claims should be equally reserved and narrow. I certainly wouldn't start invoking morality, judgment, or other abstractions of the human psyche (as did the author of this article) from such a small and contrived set of experiments.
- lutusp 14y ago
  > The study is not impressive. Most studies aren't, and exploratory, early-stage studies rarely can be. Yes, but a longitudinal examination of psychology publications shows that, compared to mainstream science, psychology studies are rarely anything but "early-stage studies". Almost none of them is ever successfully replicated, nor is an effort made to replicate them, compared to other fields. > I certainly wouldn't start invoking morality, judgment, or other abstractions of the human psyche (as did the author of this article) from such a small and contrived set of experiments. I of course agree, but then it's a psychology study, where such hand-waving claims are commonplace.
redcircle 14y ago
1. It has lots of meaning before replication, which is why people tend to replicate things --- they were inspired. 2. Apparently the first study isn't science. I didn't realize that science was a catch-22 situation. 3. This is so absurd that I feel like I'm being trolled. They have a testable hypothesis: that acetaminophen/paracetamol has a side effect of X. 4. See https://yourlogicalfallacyis.com/ https://yourlogicalfallacyis.com/
- lutusp 14y ago
  > 1. It has lots of meaning before replication ... Nothing in science has meaning before replication. Look at cold fusion for a classic example where this rule was ignored. Of course, you may be speaking of the prevailing standards in psychology, in which case you're right -- but then psychology isn't a science. > 2. Apparently the first study isn't science. That's correct -- it isn't. > I didn't realize that science was a catch-22 situation. Only if you can't replicate any studies. That's psychology's problem, but not science's problem. In science, studies are replicated regularly, and no one takes a study seriously if it hasn't been replicated, especially where human health is concerned. > They have a testable hypothesis: that acetaminophen/paracetamol has a side effect of X. That is not an explanation, it's a description. Science requires testable explanations. The paper doesn't presume to explain its results. If I say "the night sky is full of little points of light", that's certainly testable -- someone else can go outside the tent and confirm it, but it's just a description -- not a basis for science, which requires that someone have the nerve to suggest an explanation for what has been described. But if I say "those points of light are actually thermonuclear furnaces like our sun, at greater distances", I have offered a testable, falsifiable explanation. That's science. If someone correlated a specific biochemical action in individual neurons with later very specific behavior, that might count as science, but that would be neuroscience, not psychology. Psychology doesn't try to analytically connect specific physical causes to effects -- it's satisfied to describe outcomes without pretending to know why they are so. > See https://yourlogicalfallacyis.com/ https://yourlogicalfallacyis.com/ The fallacy is yours, not mine. You appear to think one study making ten claims constitutes science. In fact, science is ten studies making one claim.
  - 14y ago
jleader 14y ago
Your #1 is false. A single unreplicated study is much less meaningful than several successful replications, but it does have some meaning: "this result looks interesting, someone should try to replicate it and expand it". If the the first study on any question is completely meaningless, there'd be no motivation to try to replicate it. Regarding #2, it depends what you mean by "making a claim". A study could test a single hypothesis, validate the hypothesis, and then list 10 things that arguably could follow from that hypothesis. For example, after timing the transit of Mercury and validating relativity, a paper could then in its conclusion list other likely consequences of relativity. I think a lot of your statements about "science is ..." are confounding a single study with the entire body of scientific progress. Science requires replication and science requires theoretical explanations. That doesn't mean that a single study always has to provide its own replication (it can't!) or theoretical explanation. An experiment that dis-confirms a theory doesn't have to provide an alternative theory in order to be part of science; the theory may come later.
- lutusp 14y ago
  > Your #1 is false. A single unreplicated study is much less meaningful than several successful replications, but it does have some meaning ... Not to a scientist. Without replication, a single study is conjecture, interesting but weightless. Are you not aware that an unreplicated study is a standing joke among scientists? Why do you think the gag science journal is named the "Journal of Irreproducibe Results"? Scientists find the name funny because (a) it's an inside joke and (b) it's a truth about science that member of the public just don't get. http://www.jir.com/ http://www.jir.com/ > That doesn't mean that a single study always has to provide its own replication Then it's a good thing I never said that. But a single study has no standing until and unless a disinterested, independent laboratory succeeds in replicating it. This issue is so critical to the future of psychology that, after a series of scandals about replication and credibility, a well-known figure in the field has made an emergency call to create replication circles among psychologists to (a) replicate each other's studies, and thereby try to (b) restore public confidence in the field. He says without a replication program, "I see a train wreck looming." http://www.nature.com/polopoly_fs/7.6716.1349271308!/suppinfoFile/Kahneman%20Letter.pdf http://www.nature.com/polopoly_fs/7.6716.1349271308!/suppinf... (PDF) Quote: "For all these reasons, right or wrong, your field is now the poster child for doubts about the integrity of psychological research. Your problem is not with the few people who have actively challenged the validity of some priming results. It is with the much larger population of colleagues who in the past accepted your surprising results as facts when they were published. ... My reason for writing this letter is that I see a train wreck looming." To see the importance of replication, all you need to do is read the scientific literature.
darkarmani 14y ago
> This study has precisely no meaning until it has been replicated Why would one replicate it if it has no meaning? Might as well try a new experiment.
D_Alex 14y ago
Nice, this needed to be said, have an upvote. But let me talk about point 3. Science has many parts, one large part is gathering observations without even trying to explain - eg. look at the work of 19th century naturalists, who travelled the world collecting drawings of local flora and fauna. Another part is formulating and testing hypotheses based on casual observation which are not yet supported by a theory, and do not really have explanatory powers eg. black men have larger... nostrils. Quite often an explanation for a hypothesis is found long after the hypothesis itself has been proven correct. Eg: "puerperal fever is a contagious disease (and washing hands helps prevent its transmission)" has been establishd long before the germ theory of disease was even formed.
- lutusp 14y ago
  > Science has many parts, one large part is gathering observations without even trying to explain ... Yes, and science requires microscopes and telescopes. But science isn't defined by microscopes and telescopes, it's defined by what scientists do with them. By themselves, these individual elements aren't science, because they don't lead to a falsifiable proposed explanation, the single essential element in the definition of science. http://en.wikipedia.org/wiki/Falsifiability http://en.wikipedia.org/wiki/Falsifiability Quote: "Simply, to be scientific, a theory must predict at least some observation potentially refutable by observation." > Quite often an explanation for a hypothesis is found long after the hypothesis itself has been proven correct ... A hypothesis cannot be "proven correct". That's reserved to theories -- explanations that generalize observations and predict their relevance to phenomena not yet examined. > "puerperal fever is a contagious disease (and washing hands helps prevent its transmission)" has been establishd long before the germ theory of disease was even formed. Yes, and shaking a dried gourd over a cold sufferer will cure his cold. Isn't that science? -- it always works! If I visit 1000 cold sufferers and shake my gourd over them, every single one of them will get better if I shake the gourd long enough. So is my procedure science? No, it is not -- because I have not tried to explain why my method works, nor have I considered alternative explanations for my result. The dried gourd example, and a thousand other examples of sloppy thinking, show that testable, falsifiable explanations are a requirement for science.
vacri 14y ago
Psychology is a scientific discipline, and it's clear you have a chip on your shoulder about the field.
- lutusp 14y ago
  > Psychology is a scientific discipline Thomas Insel, the sitting chair of the NIMH, disagrees with you. In his Scientific American article "Faulty Circuits" (http://www.scientificamerican.com/article.cfm?id=faulty-circuits http://www.scientificamerican.com/article.cfm?id=faulty-circ...), he says: "In most areas of medicine, doctors have historically tried to glean something about the underlying cause of a patient’s illness before figuring out a treatment that addresses the source of the problem. When it came to mental or behavioral disorders in the past, however, no physical cause was detectable so the problem was long assumed by doctors to be solely “mental,” and psychological therapies followed suit. Today scientific approaches based on modern biology, neuroscience and genomics are replacing nearly a century of purely psychological theories, yielding new approaches to the treatment of mental illnesses." I draw your attention to the phrase "scientific approaches based on modern biology, neuroscience and genomics are replacing nearly a century of purely psychological theories ..." Science requires empirically testable, falsifiable theories. Psychology doesn't have empirically testable, falsifiable theories. End of story.

by lutusp 14y ago

carbocation 14y ago

I was going to write in disagreement with your point #3, but then I went back and re-read the Atlantic article. In that context, I can see why you would write that. However, I think it needs this context, so to others who were going to knee-jerk react to point #3, re-read the article first. The study is not impressive. Most studies aren't, and exploratory, early-stage studies rarely can be. It's OK that early studies of a phenomenon are not impressive, but the tests do seem to be extremely narrow. The claims should be equally reserved and narrow. I certainly wouldn't start invoking morality, judgment, or other abstractions of the human psyche (as did the author of this article) from such a small and contrived set of experiments.

lutusp 14y ago
> The study is not impressive. Most studies aren't, and exploratory, early-stage studies rarely can be. Yes, but a longitudinal examination of psychology publications shows that, compared to mainstream science, psychology studies are rarely anything but "early-stage studies". Almost none of them is ever successfully replicated, nor is an effort made to replicate them, compared to other fields. > I certainly wouldn't start invoking morality, judgment, or other abstractions of the human psyche (as did the author of this article) from such a small and contrived set of experiments. I of course agree, but then it's a psychology study, where such hand-waving claims are commonplace.

redcircle 14y ago

1. It has lots of meaning before replication, which is why people tend to replicate things --- they were inspired. 2. Apparently the first study isn't science. I didn't realize that science was a catch-22 situation. 3. This is so absurd that I feel like I'm being trolled. They have a testable hypothesis: that acetaminophen/paracetamol has a side effect of X. 4. See https://yourlogicalfallacyis.com/ https://yourlogicalfallacyis.com/

lutusp 14y ago
> 1. It has lots of meaning before replication ... Nothing in science has meaning before replication. Look at cold fusion for a classic example where this rule was ignored. Of course, you may be speaking of the prevailing standards in psychology, in which case you're right -- but then psychology isn't a science. > 2. Apparently the first study isn't science. That's correct -- it isn't. > I didn't realize that science was a catch-22 situation. Only if you can't replicate any studies. That's psychology's problem, but not science's problem. In science, studies are replicated regularly, and no one takes a study seriously if it hasn't been replicated, especially where human health is concerned. > They have a testable hypothesis: that acetaminophen/paracetamol has a side effect of X. That is not an explanation, it's a description. Science requires testable explanations. The paper doesn't presume to explain its results. If I say "the night sky is full of little points of light", that's certainly testable -- someone else can go outside the tent and confirm it, but it's just a description -- not a basis for science, which requires that someone have the nerve to suggest an explanation for what has been described. But if I say "those points of light are actually thermonuclear furnaces like our sun, at greater distances", I have offered a testable, falsifiable explanation. That's science. If someone correlated a specific biochemical action in individual neurons with later very specific behavior, that might count as science, but that would be neuroscience, not psychology. Psychology doesn't try to analytically connect specific physical causes to effects -- it's satisfied to describe outcomes without pretending to know why they are so. > See https://yourlogicalfallacyis.com/ https://yourlogicalfallacyis.com/ The fallacy is yours, not mine. You appear to think one study making ten claims constitutes science. In fact, science is ten studies making one claim.
- 14y ago

jleader 14y ago

Your #1 is false. A single unreplicated study is much less meaningful than several successful replications, but it does have some meaning: "this result looks interesting, someone should try to replicate it and expand it". If the the first study on any question is completely meaningless, there'd be no motivation to try to replicate it. Regarding #2, it depends what you mean by "making a claim". A study could test a single hypothesis, validate the hypothesis, and then list 10 things that arguably could follow from that hypothesis. For example, after timing the transit of Mercury and validating relativity, a paper could then in its conclusion list other likely consequences of relativity. I think a lot of your statements about "science is ..." are confounding a single study with the entire body of scientific progress. Science requires replication and science requires theoretical explanations. That doesn't mean that a single study always has to provide its own replication (it can't!) or theoretical explanation. An experiment that dis-confirms a theory doesn't have to provide an alternative theory in order to be part of science; the theory may come later.

lutusp 14y ago
> Your #1 is false. A single unreplicated study is much less meaningful than several successful replications, but it does have some meaning ... Not to a scientist. Without replication, a single study is conjecture, interesting but weightless. Are you not aware that an unreplicated study is a standing joke among scientists? Why do you think the gag science journal is named the "Journal of Irreproducibe Results"? Scientists find the name funny because (a) it's an inside joke and (b) it's a truth about science that member of the public just don't get. http://www.jir.com/ http://www.jir.com/ > That doesn't mean that a single study always has to provide its own replication Then it's a good thing I never said that. But a single study has no standing until and unless a disinterested, independent laboratory succeeds in replicating it. This issue is so critical to the future of psychology that, after a series of scandals about replication and credibility, a well-known figure in the field has made an emergency call to create replication circles among psychologists to (a) replicate each other's studies, and thereby try to (b) restore public confidence in the field. He says without a replication program, "I see a train wreck looming." http://www.nature.com/polopoly_fs/7.6716.1349271308!/suppinfoFile/Kahneman%20Letter.pdf http://www.nature.com/polopoly_fs/7.6716.1349271308!/suppinf... (PDF) Quote: "For all these reasons, right or wrong, your field is now the poster child for doubts about the integrity of psychological research. Your problem is not with the few people who have actively challenged the validity of some priming results. It is with the much larger population of colleagues who in the past accepted your surprising results as facts when they were published. ... My reason for writing this letter is that I see a train wreck looming." To see the importance of replication, all you need to do is read the scientific literature.

darkarmani 14y ago

> This study has precisely no meaning until it has been replicated Why would one replicate it if it has no meaning? Might as well try a new experiment.

D_Alex 14y ago

Nice, this needed to be said, have an upvote. But let me talk about point 3. Science has many parts, one large part is gathering observations without even trying to explain - eg. look at the work of 19th century naturalists, who travelled the world collecting drawings of local flora and fauna. Another part is formulating and testing hypotheses based on casual observation which are not yet supported by a theory, and do not really have explanatory powers eg. black men have larger... nostrils. Quite often an explanation for a hypothesis is found long after the hypothesis itself has been proven correct. Eg: "puerperal fever is a contagious disease (and washing hands helps prevent its transmission)" has been establishd long before the germ theory of disease was even formed.

lutusp 14y ago
> Science has many parts, one large part is gathering observations without even trying to explain ... Yes, and science requires microscopes and telescopes. But science isn't defined by microscopes and telescopes, it's defined by what scientists do with them. By themselves, these individual elements aren't science, because they don't lead to a falsifiable proposed explanation, the single essential element in the definition of science. http://en.wikipedia.org/wiki/Falsifiability http://en.wikipedia.org/wiki/Falsifiability Quote: "Simply, to be scientific, a theory must predict at least some observation potentially refutable by observation." > Quite often an explanation for a hypothesis is found long after the hypothesis itself has been proven correct ... A hypothesis cannot be "proven correct". That's reserved to theories -- explanations that generalize observations and predict their relevance to phenomena not yet examined. > "puerperal fever is a contagious disease (and washing hands helps prevent its transmission)" has been establishd long before the germ theory of disease was even formed. Yes, and shaking a dried gourd over a cold sufferer will cure his cold. Isn't that science? -- it always works! If I visit 1000 cold sufferers and shake my gourd over them, every single one of them will get better if I shake the gourd long enough. So is my procedure science? No, it is not -- because I have not tried to explain why my method works, nor have I considered alternative explanations for my result. The dried gourd example, and a thousand other examples of sloppy thinking, show that testable, falsifiable explanations are a requirement for science.

vacri 14y ago

Psychology is a scientific discipline, and it's clear you have a chip on your shoulder about the field.

lutusp 14y ago
> Psychology is a scientific discipline Thomas Insel, the sitting chair of the NIMH, disagrees with you. In his Scientific American article "Faulty Circuits" (http://www.scientificamerican.com/article.cfm?id=faulty-circuits http://www.scientificamerican.com/article.cfm?id=faulty-circ...), he says: "In most areas of medicine, doctors have historically tried to glean something about the underlying cause of a patient’s illness before figuring out a treatment that addresses the source of the problem. When it came to mental or behavioral disorders in the past, however, no physical cause was detectable so the problem was long assumed by doctors to be solely “mental,” and psychological therapies followed suit. Today scientific approaches based on modern biology, neuroscience and genomics are replacing nearly a century of purely psychological theories, yielding new approaches to the treatment of mental illnesses." I draw your attention to the phrase "scientific approaches based on modern biology, neuroscience and genomics are replacing nearly a century of purely psychological theories ..." Science requires empirically testable, falsifiable theories. Psychology doesn't have empirically testable, falsifiable theories. End of story.