7 ms·

Many Psychology Findings Not as Strong as Claimed, Study Says

99 points by mcgwiz 11y ago

Lawtonfogle 11y ago
Perhaps the biggest take away I found while studying psychology is that a lot of research in the field just couldn't be trusted until you vetted it. Political pressures are too great a corrupting factor. Words would be redefined. Conclusions overstated or over applied. This is on top of the already existing 'publish or perish' issue that impacts science as a whole. The closer one was to neurology (like physiological psychology), the better it became. The closer one was to sociology (like IO psychology) or to a politically charged issue, the worse it became. Also applies to psychiatry. I remember reading some of the papers published related to the DSM-V and at one point it looked like little more than peer reviewed version of two siblings fighting (though that was the worse case, not the average). One big thing is to look at how the researchers defined words and look into how things translate when multiple languages were involved.
- _delirium 11y ago
  > Words would be redefined. Conclusions overstated or over applied. Even neuroscience has a hearty helping of this. The data itself is one thing, but what it's sold as showing is quite another. For example if you read a claim that neuroscience has discovered something about "addiction", "free will", "motivation", "friendship", "love", etc., dig into how they've chosen to define those terms for the purpose of the study. There's often a bit of concept-laundering going on, where a common term is defined using a very narrow (and often conveniently chosen) definition to show a result, but then the implications of that result are discussed with reference to the original, more general sense of the term. News articles about neuroscience are the worst, of course, but even quite a few scientists themselves do this.
afarrell 11y ago
Here is the paper the Reproducability Project just released on Estimating the reproducibility of psychological science: http://www.sciencemag.org/content/349/6251/aac4716 http://www.sciencemag.org/content/349/6251/aac4716
wesleytodd 11y ago
"Study says other studies are flawed".... Didn't read the article, but couldn't help but comment on the title, good laughs.
- WalterGR 11y ago
  "Many Findings About Psychology Findings Not Being as Strong as Claimed Not as Strong as Claimed, Study Says"
  - thehypotemoose 11y ago
    This chain of logic was covered in a 2011 fiction piece published in Science's competing journal Nature: http://www.nature.com/nature/journal/v477/n7363/full/477244a.html http://www.nature.com/nature/journal/v477/n7363/full/477244a... “Although it is nonsensical to rely on evidence provided by human-based research when judging whether humans are themselves inept, in doing so, the editors (all human, I note) provide a perfect example of the feebleness of human reasoning, thereby validating their claims.”
legitster 11y ago
As a marketer, I have long given up on psychology as a field. The practitioners are too wishy-washy, the experiments too prone to confirmation bias, and the results too unusable. I feel behavioral economists cover many of the same subjects, but their work is much more interesting and scientific. The experiments are tighter and ask better questions. And they focus more on specific characteristics in decision making, and less on asking big questions.
- vacri 11y ago
  > And they focus more on specific characteristics in decision making, and less on asking big questions. It seems more that it's behavioural economists that do studies relevant to your marketing, not a flaw in psychology itself. Psychology is an incredibly broad field, and it asks questions both big and small. Plenty of psychology studies ask very specific questions looking for very specific effects.
  - will_work4tears 11y ago
    My place of employment was funded a grant, and did a study to show that teens were using mobile devices more often than they did years ago (I don't remember the timeframe). I believe they got something like a 200k+ grant for this. A team of like 10 PhD psychologists to show something we already know. Not saying psychology as a field is pointless, or anything, just that they do occasionally do some silly studies.
    - superuser2 11y ago
      Therein lies the problem. On many of the questions that psychology studies, people are deeply committed to an intuitive answer. A layperson reading any psychology experiment will either say "That's obvious! You spent how much establishing something everyone already knows?" or "No way. You must be twisting the data to make a name for yourself." Particularly when it touches hot-button issues like "to what extent are [people I don't like] in control of their own situations?" You don't get that around physics quite so much.
    - 11y ago
simulate 11y ago
There has been a backlash against attempts at replication in psychology: http://phenomena.nationalgeographic.com/.../failed.../ http://phenomena.nationalgeographic.com/.../failed.../ http://andrewgelman.com/2013/12/17/replication-backlash/ http://andrewgelman.com/2013/12/17/replication-backlash/ and even a backlash to the backlash: http://phenomena.nationalgeographic.com/.../failed.../ http://phenomena.nationalgeographic.com/.../failed.../
- scott_s 11y ago
  Your last URL is the same as the first.
- Asbostos 11y ago
  One of the weird critisisms I've heard (and reflected in your first link) is "but the reproduced study had some slight difference". If a slight difference can make an effect disappear then it's not such an interesting or general effect as the original study claimed to show! Why didn't they already test for the effect of slight differences themselves?
  - lqdc13 11y ago
    You can't evaluate every single slight variation on your testing environment. That's just the nature of the field.
mamon 11y ago
Psychology is more like religion than science.
cm2187 11y ago
And of the 40 that turned the same results, a fair chunk might be just by chance.
Apes 11y ago
For anyone interested in more information on this topic, the book "Psychology Gone Wrong" by Tomasz Witkowski and Maciej Zatonski is a very in depth look into the problems in this field.
chockablock 11y ago
Here's a nice, approachable treatment of some of the ways psych studies can go wrong; it also offers concrete suggestions for fixes: Joseph P. Simmons, Leif D. Nelson, and Uri Simonsohn "False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant" Psychological Science November 2011 22: 1359-1366, Free full-text (no paywall): http://pss.sagepub.com/content/22/11/1359 http://pss.sagepub.com/content/22/11/1359 (via: http://bahanonu.com/brain/#c20150315 http://bahanonu.com/brain/#c20150315 )
hackuser 11y ago
Take the time to read the story (many comments don't reflect more than the headline), which is far more complex and interesting than the headline. For one thing, almost all studies' effects were reproduced, but they were generally weaker. * Most importantly, from the Times: Strictly on the basis of significance — a statistical measure of how likely it is that a result did not occur by chance — 35 of the studies held up, and 62 did not. (Three were excluded because their significance was not clear.) The overall “effect size,” a measure of the strength of a finding, dropped by about half across all of the studies. Yet very few of the redone studies contradicted the original ones; their results were simply weaker. * Also: The research team also measured whether the prestige of the original research group, rated by measures of expertise and academic affiliation, had any effect on the likelihood that its work stood up. It did not. * And: The only factor that did [affect the likelihood of successful reproduction] was the strength of the original effect — that is, the most robust findings tended to remain easily detectable, if not necessarily as strong. * Finally: The project’s authors write that, despite the painstaking effort to duplicate the original research, there could be differences in the design or context of the reproduced work that account for the different findings. Many of the original authors certainly agree. * According to several experts, there is no reason to think the problems are confined to psychology, and it could be worse in other fields. The researchers chose psychology merely because that is their field of expertise. * I haven't seen anything indicating the 100 studies are a representative sample of the population of published research, and at least one scientist raised this question.
- conistonwater 11y ago
  > The overall “effect size,” a measure of the strength of a finding, dropped by about half across all of the studies. In general, if you conduct an experiment, then conditional on finding significance the estimate of the effect size is going to be an overestimate of the true effect size. This is because the data that was extreme enough to produce a low p-value is also likely to be more extreme than the true population. Obviously this effect is less important when the true effect size is large enough - in high-powered studies. So this might not actually be so important: it may also mean that the effect size estimated in the original paper was simply miscalculated and estimated to be too high.
- 11y ago
RA_Fisher 11y ago
This is how I evaluate studies, "are the data and code made available?" It's a simple request and this study is the first I've ever seen offer both. Bravo!
martingoodson 11y ago
The replication effect sizes are almost all positive [1]. This study represents very strong validation that the published results were qualitatively true, in the main. The regression to the mean effect is unsurprising and doesn't diminish this finding. Given the difficulty of performing this kind of research, this is a very positive result for the field. [1]http://m.sciencemag.org/content/349/6251/aac4716/F1.expansion.html http://m.sciencemag.org/content/349/6251/aac4716/F1.expansio...

99 points by mcgwiz 11y ago

Perhaps the biggest take away I found while studying psychology is that a lot of research in the field just couldn't be trusted until you vetted it. Political pressures are too great a corrupting factor. Words would be redefined. Conclusions overstated or over applied. This is on top of the already existing 'publish or perish' issue that impacts science as a whole. The closer one was to neurology (like physiological psychology), the better it became. The closer one was to sociology (like IO psychology) or to a politically charged issue, the worse it became. Also applies to psychiatry. I remember reading some of the papers published related to the DSM-V and at one point it looked like little more than peer reviewed version of two siblings fighting (though that was the worse case, not the average). One big thing is to look at how the researchers defined words and look into how things translate when multiple languages were involved.

_delirium 11y ago
> Words would be redefined. Conclusions overstated or over applied. Even neuroscience has a hearty helping of this. The data itself is one thing, but what it's sold as showing is quite another. For example if you read a claim that neuroscience has discovered something about "addiction", "free will", "motivation", "friendship", "love", etc., dig into how they've chosen to define those terms for the purpose of the study. There's often a bit of concept-laundering going on, where a common term is defined using a very narrow (and often conveniently chosen) definition to show a result, but then the implications of that result are discussed with reference to the original, more general sense of the term. News articles about neuroscience are the worst, of course, but even quite a few scientists themselves do this.

afarrell 11y ago

Here is the paper the Reproducability Project just released on Estimating the reproducibility of psychological science: http://www.sciencemag.org/content/349/6251/aac4716 http://www.sciencemag.org/content/349/6251/aac4716

wesleytodd 11y ago

"Study says other studies are flawed".... Didn't read the article, but couldn't help but comment on the title, good laughs.

WalterGR 11y ago
"Many Findings About Psychology Findings Not Being as Strong as Claimed Not as Strong as Claimed, Study Says"
- thehypotemoose 11y ago
  This chain of logic was covered in a 2011 fiction piece published in Science's competing journal Nature: http://www.nature.com/nature/journal/v477/n7363/full/477244a.html http://www.nature.com/nature/journal/v477/n7363/full/477244a... “Although it is nonsensical to rely on evidence provided by human-based research when judging whether humans are themselves inept, in doing so, the editors (all human, I note) provide a perfect example of the feebleness of human reasoning, thereby validating their claims.”

legitster 11y ago

As a marketer, I have long given up on psychology as a field. The practitioners are too wishy-washy, the experiments too prone to confirmation bias, and the results too unusable. I feel behavioral economists cover many of the same subjects, but their work is much more interesting and scientific. The experiments are tighter and ask better questions. And they focus more on specific characteristics in decision making, and less on asking big questions.

vacri 11y ago
> And they focus more on specific characteristics in decision making, and less on asking big questions. It seems more that it's behavioural economists that do studies relevant to your marketing, not a flaw in psychology itself. Psychology is an incredibly broad field, and it asks questions both big and small. Plenty of psychology studies ask very specific questions looking for very specific effects.
- will_work4tears 11y ago
  My place of employment was funded a grant, and did a study to show that teens were using mobile devices more often than they did years ago (I don't remember the timeframe). I believe they got something like a 200k+ grant for this. A team of like 10 PhD psychologists to show something we already know. Not saying psychology as a field is pointless, or anything, just that they do occasionally do some silly studies.
  - superuser2 11y ago
    Therein lies the problem. On many of the questions that psychology studies, people are deeply committed to an intuitive answer. A layperson reading any psychology experiment will either say "That's obvious! You spent how much establishing something everyone already knows?" or "No way. You must be twisting the data to make a name for yourself." Particularly when it touches hot-button issues like "to what extent are [people I don't like] in control of their own situations?" You don't get that around physics quite so much.
  - 11y ago

simulate 11y ago

There has been a backlash against attempts at replication in psychology: http://phenomena.nationalgeographic.com/.../failed.../ http://phenomena.nationalgeographic.com/.../failed.../ http://andrewgelman.com/2013/12/17/replication-backlash/ http://andrewgelman.com/2013/12/17/replication-backlash/ and even a backlash to the backlash: http://phenomena.nationalgeographic.com/.../failed.../ http://phenomena.nationalgeographic.com/.../failed.../

scott_s 11y ago
Your last URL is the same as the first.
Asbostos 11y ago
One of the weird critisisms I've heard (and reflected in your first link) is "but the reproduced study had some slight difference". If a slight difference can make an effect disappear then it's not such an interesting or general effect as the original study claimed to show! Why didn't they already test for the effect of slight differences themselves?
- lqdc13 11y ago
  You can't evaluate every single slight variation on your testing environment. That's just the nature of the field.

mamon 11y ago

Psychology is more like religion than science.

cm2187 11y ago

And of the 40 that turned the same results, a fair chunk might be just by chance.

Apes 11y ago

For anyone interested in more information on this topic, the book "Psychology Gone Wrong" by Tomasz Witkowski and Maciej Zatonski is a very in depth look into the problems in this field.

chockablock 11y ago

Here's a nice, approachable treatment of some of the ways psych studies can go wrong; it also offers concrete suggestions for fixes: Joseph P. Simmons, Leif D. Nelson, and Uri Simonsohn "False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant" Psychological Science November 2011 22: 1359-1366, Free full-text (no paywall): http://pss.sagepub.com/content/22/11/1359 http://pss.sagepub.com/content/22/11/1359 (via: http://bahanonu.com/brain/#c20150315 http://bahanonu.com/brain/#c20150315 )

hackuser 11y ago

Take the time to read the story (many comments don't reflect more than the headline), which is far more complex and interesting than the headline. For one thing, almost all studies' effects were reproduced, but they were generally weaker. * Most importantly, from the Times: Strictly on the basis of significance — a statistical measure of how likely it is that a result did not occur by chance — 35 of the studies held up, and 62 did not. (Three were excluded because their significance was not clear.) The overall “effect size,” a measure of the strength of a finding, dropped by about half across all of the studies. Yet very few of the redone studies contradicted the original ones; their results were simply weaker. * Also: The research team also measured whether the prestige of the original research group, rated by measures of expertise and academic affiliation, had any effect on the likelihood that its work stood up. It did not. * And: The only factor that did [affect the likelihood of successful reproduction] was the strength of the original effect — that is, the most robust findings tended to remain easily detectable, if not necessarily as strong. * Finally: The project’s authors write that, despite the painstaking effort to duplicate the original research, there could be differences in the design or context of the reproduced work that account for the different findings. Many of the original authors certainly agree. * According to several experts, there is no reason to think the problems are confined to psychology, and it could be worse in other fields. The researchers chose psychology merely because that is their field of expertise. * I haven't seen anything indicating the 100 studies are a representative sample of the population of published research, and at least one scientist raised this question.

conistonwater 11y ago
> The overall “effect size,” a measure of the strength of a finding, dropped by about half across all of the studies. In general, if you conduct an experiment, then conditional on finding significance the estimate of the effect size is going to be an overestimate of the true effect size. This is because the data that was extreme enough to produce a low p-value is also likely to be more extreme than the true population. Obviously this effect is less important when the true effect size is large enough - in high-powered studies. So this might not actually be so important: it may also mean that the effect size estimated in the original paper was simply miscalculated and estimated to be too high.
11y ago

RA_Fisher 11y ago

This is how I evaluate studies, "are the data and code made available?" It's a simple request and this study is the first I've ever seen offer both. Bravo!

martingoodson 11y ago

The replication effect sizes are almost all positive [1]. This study represents very strong validation that the published results were qualitatively true, in the main. The regression to the mean effect is unsurprising and doesn't diminish this finding. Given the difficulty of performing this kind of research, this is a very positive result for the field. [1]http://m.sciencemag.org/content/349/6251/aac4716/F1.expansion.html http://m.sciencemag.org/content/349/6251/aac4716/F1.expansio...