8 ms·

We found only one-third of published psychology research is reliable – now what?

59 points by tempestn 11y ago

deleted 11y ago
[deleted]
douche 11y ago
Reach for an extra-large grain of salt whenever the nightly news trumpets some new breakthrough in the psychological field. Continue avoiding psychologists whenever possible.
vox_mollis 11y ago
Sneaking suspicion that this will readily extend to other soft sciences, particularly sociology. A good chunk of our social and educational policy is informed by psychology and sociology. Since we have all grown up within various systems and institutions that have been largely informed by bad research, it's unlikely much will happen, since we see the claims made by these charlatans to be normal or common sense. Nonetheless, it will be amusing watching the charlatans try to defend themselves in the coming years.
- unprepare 11y ago
  This seems to me to be a clear cut case of poor incentives in the academic research sector. No graduate student is incentivized to re perform existing experiments to verify them, no they find much more success in creating a unique experiment which tests a previously untested hypothesis. This is modern academic science. If you aren't doing something newsworthy, you're not getting any funding.
  - EdiX 11y ago
    >No graduate student is incentivized to re perform existing experiments to verify them If you look at how this thing with psychology started ("replication bullies") or at what Brockman said about the LaCour scandal earlier this year (he didn't pursue publication initially because he thought he'd risk his career), it's not that they aren't incentivized, it's that they are actively discouraged.
  - Asbostos 11y ago
    Not only is lack of replication bad, but I think also excess of new research is bad too. Everybody makes their own unique study, too different to compare to existing research, too small a sample size to be reliable, too singular among thousands of equally unimportant and unreliable studies that it goes unnoticed and unused. Really, those researchers trying to do something novel but being incompetent at it, should put their skills to use replicating something else instead. Sure this would lead to them being seen as 2nd class scientists. But we still need that service, just as we still need cleaners even though they're seen as 2nd class workers. Universities are the ones providing the incentives. Their management should be stopping and thinking about if their policies are adding value to scientific knowledge or subtracting it.
- omginternets 11y ago
  >particularly sociology I'll take it a step further. The reason we're hearing about psychology and not sociology is that few people take sociology seriously, as a science. Sociology is a science like political science is a science in that any irrefutable claim constitutes a valid theory. Evidence is secondary and the notion of falsifiability doesn't even come up. Psychology has a very hard-science core, whose reputation is so tarnished by its "social" counterpart that no self-respecting researcher wants to be called a psychologist anymore. Replace your searches for psychology with cognitive science and you'll find a lot of good, reproducible, research such as this: http://cavlab.net/?lang=en http://cavlab.net/?lang=en
mc_tryhard 11y ago
You could tell me what this is: http://imgur.com/HLAfj64 http://imgur.com/HLAfj64 ... Specifically the coil array is a WTF. Coil array is for parallel MRI via only RF [radio] - University of Colorado - google I studied mercury for about a month and was horrified by some facts. Specifically it was known to cause mental illness since 500BC, and how to make a mercury mine more profitable than a gold mine: https://voat.co/v/politics/comments/171588 https://voat.co/v/politics/comments/171588 causation: The problem with mercury is its a radio/electric conductor. Innoculations used since 1700 to prevent disease. Mercury added to vaccine in 1920. Discovered in 1930 and 1960 Brain Axons (nerves) use electrical impulse. Mercury could create a literal short circuit. image explantation: http://imgur.com/B9OVJOY http://imgur.com/B9OVJOY Mercury (vaccines) was known to cause mental illness, breathing disorder, and heavy metal toxicity since 500 BC [dartmouth tox history mercury] Mercury is a metal, and is excitable by radio. radio is a form of radiation. the earth is a big ball of radiation. therefore the mercury will always be excited. ------------ You could try shielding: EMF sensitivity is a thing. My schizophrenia gone for $1 of foil and earplugs. Mylar blankets ($3 from amazon) could block light(laser), emf (magnets), microwave radio and radar Air ionizers or orgonite or negative ion projectors could block negative ions Earplugs and stuff will block low frequency sound (mind control) and the freq men make ----------- We should ask why the previous treatments for mental illness were barbaric [US superme court testimonies] Lithium (a poison so toxic its used to kill unkillable cancer cells) Shock torture Lobotomy Hand crank drilling a hole in the skull to drain blood -------------- We should ask why theres so much chemicals in the food: https://medium.com/@bananaguy80/the-great-filter-theory-are-we-a-filter-f73a2f1e933f https://medium.com/@bananaguy80/the-great-filter-theory-are-... We should ask why the newspapers silence this stuff. Why my congress men has an unlisted email, his contact form is broken, doesnt answer emails, facebook etc. And then get guns for your bull E.T.s.
omginternets 11y ago
I actually have an answer for this: Stop talking about "psychology" like it's a monolithic thing. Start by separating "hard" psychology (perception, attention, psychophysics, etc) from social psychology. Papers about saccadic adaptation are, I suspect, much more reproducible than those about implicit association.
- simonster 11y ago
  The OSC Science paper separates cognitive and social psychology papers and journals, and cognitive psychology papers replicate at about twice the rate of social psychology papers. But still only a bit more than half of the cognitive papers replicate, so I think there are still deeper issues.
  - hga 11y ago
    But still only a bit more than half of the cognitive papers replicate, so I think there are still deeper issues. That puts them in the general level of quality for all bio-medical research, so, yeah, deeper issues indeed, but I find it fascinating that this sub-group is so good.
    - omginternets 11y ago
      This because that sub-group is very, very close to medical research. The problem with social psychology, on the whole, is that it's highly politicized. This is especially true with regards to any study involving race, gender, or "identity" (in the sense of identity politics).
- Lawtonfogle 11y ago
  One thing I noticed is that the more focused the subset of psychology, the better the results. For example psychology that looks at a group of neurons and how they function is vastly better than psychology looking at how some early life event X leads to later problems in life.
  - 11y ago
analog31 11y ago
One small suggestion: Eliminate "published and outa here," the policy where students have to publish their MS and Phd work in order to receive those degrees. All that does is to fill the journals with crap.
- omginternets 11y ago
  I agree in principle but oftentimes the research done by grad-students is part of a project that is spearheaded by a career researcher. I question the notion that MS and PhD work is filling journals with crap. Instead, I think there are three problems: 1. The inherent problem of inference. When you set p < .05, then 5% of studies will yield false positives. This can be mitigated by a variety of statistical approaches, but it's an irreducible problem on a fundamental level. 2. Certain (sub)disciplines of science aren't very scientific. In particular, they suffer from a high incidence of unfalsifiable claims and hand-wavy definitions. A good example, I think, is the shame vs guilt literature. 3. Certain (sub)disciplines of science are highly politicized. I point my finger at anything involving "identity" (in the sense of "identity politics"), "diversity" (sex, age, ethnicity, sexual orientation, etc) or "bias". Regarding point 3, take a look at the faculty and students of any university you'd like and take stock of how many Black people are working on racism, women on sexism, LGBT on sexual orientation, etc. As a cognitive neuroscientist, I don't want to claim that people can't study themselves, but in light of the serious reproducibility issues that social psychology is facing, I think this correlation is anything but trivial. I strongly suspect that the "hard psych vs social psych" dichotomy can be further broken down, and that the sub-disciplines listed in point 3 account for a disproportionate quantity of false-positive results, on accounts of flawed methodology. I suspect that much of this research is actually an attempt to validate political opinion. I may be wrong about all of this, but at least the core claim testable with a handful of linear regressions.
  - analog31 11y ago
    All good stuff. Thanks. I'm just saying, don't make publication a requirement for graduation. I'm not a statistician, but as I understand it, a problem with the p value is that the standard calculation hinges on assumptions that are not necessarily met by the data, and if those assumptions were correctly accounted for, p would be higher. If p < 0.05, and in practice 50% of published results are bunk, then regardless of the underlying math, a practical rule of thumb seems to be: Multiply p by 10.
medymed 11y ago
Redistribute most of the $5 billion dollars of psychology and behavioral-related NIH research funding to historically high ROI areas like infectious disease research. http://report.nih.gov/categorical_spending.aspx http://report.nih.gov/categorical_spending.aspx
DennisP 11y ago
A friend of mine is a social psychology researcher, who's published work destroying the statistics in other papers. He says this claim of unreliability is a lot more dubious than the media portrays. In many cases, they say a finding was not reproduced simply because the new study didn't quite meet the cutoff for "significance." However, if you do a meta-analysis of both studies, you find that the likelihood of the effect being real is higher, rather than lower. In other cases, the replications were done poorly. The first step of many studies is calibration. You do tests to find out what sort of things your subjects are familiar with, and calibrate questions accordingly. A common mistake in the replications was to skip the calibration step, and simply reuse the questions on subjects with different backgrounds.
- Asbostos 11y ago
  So we need the original authors to come back with their own re-replication. Instead of throwing their published paper over the wall and moving on, they are the people in the best position to actually defend it and address problems with replications. This publish-and-forget behavior annoys me about researchers. Sometimes I find results that look like they'll useful for my (non-academic) work, but the paper containing them is the final say on the matter. There's no version 2, there's no other work improving on it. It's just a dead end.
subliminalzen 11y ago
It's not just psychological research that is deeply flawed. Only a quarter of scientific drug research is successfully reproduced as well.[1] Carl Jung claimed one of the chief factors responsible for mass brainwashing is scientific rationality.[2] Society worships the Goddess of Reason while frowning down on "irrational" and non-verifiable religious testimony. Now that science is proven to be systemically corrupt, what will "rational" people base their understanding on? Aside: I designed a personality test / psychoanalytical tool that was inspired by Carl Jung. It's called Critical Stimulus and it can be found at https://www.thegamecrafter.com/games/critical-stimulus https://www.thegamecrafter.com/games/critical-stimulus. The printable version can be downloaded at gumroad: https://gumroad.com/l/criticalstimulus https://gumroad.com/l/criticalstimulus . [1] http://www.economist.com/news/briefing/21588057-scientists-think-science-self-correcting-alarming-degree-it-not-trouble http://www.economist.com/news/briefing/21588057-scientists-t... [2] https://en.wikipedia.org/wiki/Self_in_Jungian_psychology https://en.wikipedia.org/wiki/Self_in_Jungian_psychology
- omginternets 11y ago
  >Now that science is proven to be systemically corrupt, what will "rational" people base their understanding on? This is not a new notion in the least. The "rational" people will continue doing what they've always done: revising their conclusions. Science is a process, and while we can question the notion of "scientific progress" in philosophical terms, this is neither a new idea nor evidence that science doesn't work. It's certainly not evidence that science is no better than irrational thinking. >Carl Jung claimed one of the chief factors responsible for mass brainwashing is scientific rationality. Few people take psychoanalysts seriously these days, in large part because of their long record of absurd claims and shoddy clinical work. Jungian theory has it's place in a conversation about literary theory, but not in a conversation about science.
  - deleted 11y ago
    [deleted]
  - drumdance 11y ago
    I don't know about psychoanalysis as defined by Freud and Jung, but therapy in general is an intensely personal experience. I'm sure there are scientific aspects that can be brought to bear on a situation, but for a lot of people the "literary theory" part is just as helpful. Especially when you stop to think how much neurosis is fueled by pop culture (i.e. status envy).
rjruxhchd 11y ago
Far more worrying is that only a third of economics findings can be replicated without the author's help, and yet these are the supposedly scientific findings our politicians insist on basing policy off of. Even if policy seems counterintuitive and clearly harmful to the general public, we're given the explanation "because economics says so." It's clear now from the Federal Reserve replicability study that we're being duped. http://www.federalreserve.gov/econresdata/feds/2015/files/2015083pap.pdf http://www.federalreserve.gov/econresdata/feds/2015/files/20...
- eru 11y ago
  And if I read that right, they were just trying to run the code provided with the papers again.
- chongli 11y ago
  Economists are the modern equivalent to the oracles at Delphi. They speak in gibberish and the priesthood (politicians) make the interpretations which coincidentally happen to favour themselves and their friends.
  - deleted 11y ago
    [deleted]
  - sopooneo 11y ago
    I feel there's truth to that, but most at the edges. At core there is stuff economists tend to very widely agree on. There was a Planet Money episode outlining some pretty drastic policy changes that a panel of economists across the political spectrum all stood behind. Here's a write up: http://www.npr.org/sections/money/2012/07/19/157047211/six-policies-economists-love-and-politicians-hate http://www.npr.org/sections/money/2012/07/19/157047211/six-p...
geomark 11y ago
Let's be clear on the differencen between reproducible and replication. Reproducible is when an experiment's data and methods are published and other researchers can reproduce the same statistical results. An experiment that has been replicated is a new experiment that applies the same methods to a new set of subjects and arrives at statistically equivalent results. The distinction is important. Reproducible research is a lot easier but there are still famous examples of attempts to reproduce published results that revealed serious flaws. Cases where the data are kept secret to thwart reproducibility are thoroughly suspect. Replication is a harder problem, both in being able to perform a potentially costly and difficult experiment, and in the statistical analysis to show equivalent results.
- nonbel 11y ago
  >"Let's be clear on the difference between reproducible and replication." I did not find this clear. What is the distinction you are trying to make?
  - geomark 11y ago
    Reproduce: take the published data and methods, rerun the analysis, see if you get the same results. No new experiment. Replicate: perform a new experiment with new subjects, apply the same methods, see if you get statistically equivalent results.
    - nonbel 11y ago
      Thanks, I had not heard that distinction in terms before. I suppose it would be good to have, but I would not assume people are using them in that way.
      - geomark 11y ago
        Some people are making a lot of noise about making the distinction, like the guys teaching data science at John Hopkins. It's because both are important but they are distinctly different activities.
nonbel 11y ago
Now what? You need to attempt independent replications of every published claim going forward. It is clear a single published result is unreliable. This is well known but apparently needed to be rediscovered by those who misunderstand the meaning of a p-value. >"The first is a p-value, which estimates the probability that the result was arrived at purely by chance and is a false positive. (Technically, the p-value is the chance that the result, or a stronger result, would have occurred even when there was no real effect.) Generally, if a statistical test shows that the p-value is lower than 5%, the study’s results are considered “significant” – most likely due to actual effects." No, there is endless literature on this. Just to start: http://andrewgelman.com/2015/07/21/a-bad-definition-of-statistical-significance-from-the-u-s-department-of-health-and-human-services-effective-health-care-program/ http://andrewgelman.com/2015/07/21/a-bad-definition-of-stati...
- kyllo 11y ago
  You need to attempt independent replications of every published claim going forward. Taking this opportunity to plug an open source application that I work on, for cataloging and verifying independent replications, called Curate Science: https://www.curatescience.org/ https://www.curatescience.org/
- geomark 11y ago
  More to the point is how "replication" is defined. This quote covers it: "Note that the 36% figure comes from a definition of replication that mimics the definition used by regulatory agencies: results are considered replicated if a p-value < 0.05 was reached in both the original study and the replicated one." If you read the post where that quote comes from [1] they make a number of points about how a better and more rigorous definition of replicated is needed, because p-value alone doesn't tell the whole story. [1] http://simplystatistics.org/2015/10/20/we-need-a-statistically-rigorous-and-scientifically-meaningful-definition-of-replication/ http://simplystatistics.org/2015/10/20/we-need-a-statistical...
  - DennisP 11y ago
    So say one study has a p-value of .04, meaning there's a .04 probability that there's no effect and the results occurred by chance. A replication comes along and gets a p-value of .06, so it gets counted as a failed replication. And yet, the probability that both results happened by chance is only .0024.
matthferguson 11y ago
now what ... Accurately label psychology and the social sciences as philosophy not science.
millermp12 11y ago
When the science goes against the Narrative, do what our church fathers did: go with Scripture. First, the American Anthropological Association, next the APA? They might as well come of the closet and stop pretending so I can stop putting dick quotes around social "scientists." https://www.insidehighered.com/news/2010/11/30/anthroscience https://www.insidehighered.com/news/2010/11/30/anthroscience
themetrician 11y ago
The reliability of Psychology throughout the 20th century is covered in depth in the book The Culture of Critique, where the exact persons responsible for fostering in Psychology an anti-Science environment are discussed.
gregn 11y ago
From 10 years of working as a sysadmin in a Psych department, one constant in life I've learned is that Psychologists are the most incompetent, fanciful egoists in all the sciences. They have no idea what is going on basically. Or, put as a friend of mine said, "People who go into Psychology are precisely those that do not intuitively understand it already." Implying that most of us have a decent, commonsense appraisal of how things work, but psych people are the most clueless of the clueless, that's why they go into a field (first falling for its false promises) that professes to explain life's mysteries to them. None of this is surprising, except perhaps that they scored so far away from the median.
MollyR 11y ago
This is quite frightening considering how people try to set government policies based on them.

59 points by tempestn 11y ago

deleted 11y ago

[deleted]

Reach for an extra-large grain of salt whenever the nightly news trumpets some new breakthrough in the psychological field. Continue avoiding psychologists whenever possible.

vox_mollis 11y ago

Sneaking suspicion that this will readily extend to other soft sciences, particularly sociology. A good chunk of our social and educational policy is informed by psychology and sociology. Since we have all grown up within various systems and institutions that have been largely informed by bad research, it's unlikely much will happen, since we see the claims made by these charlatans to be normal or common sense. Nonetheless, it will be amusing watching the charlatans try to defend themselves in the coming years.

unprepare 11y ago
This seems to me to be a clear cut case of poor incentives in the academic research sector. No graduate student is incentivized to re perform existing experiments to verify them, no they find much more success in creating a unique experiment which tests a previously untested hypothesis. This is modern academic science. If you aren't doing something newsworthy, you're not getting any funding.
- EdiX 11y ago
  >No graduate student is incentivized to re perform existing experiments to verify them If you look at how this thing with psychology started ("replication bullies") or at what Brockman said about the LaCour scandal earlier this year (he didn't pursue publication initially because he thought he'd risk his career), it's not that they aren't incentivized, it's that they are actively discouraged.
- Asbostos 11y ago
  Not only is lack of replication bad, but I think also excess of new research is bad too. Everybody makes their own unique study, too different to compare to existing research, too small a sample size to be reliable, too singular among thousands of equally unimportant and unreliable studies that it goes unnoticed and unused. Really, those researchers trying to do something novel but being incompetent at it, should put their skills to use replicating something else instead. Sure this would lead to them being seen as 2nd class scientists. But we still need that service, just as we still need cleaners even though they're seen as 2nd class workers. Universities are the ones providing the incentives. Their management should be stopping and thinking about if their policies are adding value to scientific knowledge or subtracting it.
omginternets 11y ago
>particularly sociology I'll take it a step further. The reason we're hearing about psychology and not sociology is that few people take sociology seriously, as a science. Sociology is a science like political science is a science in that any irrefutable claim constitutes a valid theory. Evidence is secondary and the notion of falsifiability doesn't even come up. Psychology has a very hard-science core, whose reputation is so tarnished by its "social" counterpart that no self-respecting researcher wants to be called a psychologist anymore. Replace your searches for psychology with cognitive science and you'll find a lot of good, reproducible, research such as this: http://cavlab.net/?lang=en http://cavlab.net/?lang=en

mc_tryhard 11y ago

You could tell me what this is: http://imgur.com/HLAfj64 http://imgur.com/HLAfj64 ... Specifically the coil array is a WTF. Coil array is for parallel MRI via only RF [radio] - University of Colorado - google I studied mercury for about a month and was horrified by some facts. Specifically it was known to cause mental illness since 500BC, and how to make a mercury mine more profitable than a gold mine: https://voat.co/v/politics/comments/171588 https://voat.co/v/politics/comments/171588 causation: The problem with mercury is its a radio/electric conductor. Innoculations used since 1700 to prevent disease. Mercury added to vaccine in 1920. Discovered in 1930 and 1960 Brain Axons (nerves) use electrical impulse. Mercury could create a literal short circuit. image explantation: http://imgur.com/B9OVJOY http://imgur.com/B9OVJOY Mercury (vaccines) was known to cause mental illness, breathing disorder, and heavy metal toxicity since 500 BC [dartmouth tox history mercury] Mercury is a metal, and is excitable by radio. radio is a form of radiation. the earth is a big ball of radiation. therefore the mercury will always be excited. ------------ You could try shielding: EMF sensitivity is a thing. My schizophrenia gone for $1 of foil and earplugs. Mylar blankets ($3 from amazon) could block light(laser), emf (magnets), microwave radio and radar Air ionizers or orgonite or negative ion projectors could block negative ions Earplugs and stuff will block low frequency sound (mind control) and the freq men make ----------- We should ask why the previous treatments for mental illness were barbaric [US superme court testimonies] Lithium (a poison so toxic its used to kill unkillable cancer cells) Shock torture Lobotomy Hand crank drilling a hole in the skull to drain blood -------------- We should ask why theres so much chemicals in the food: https://medium.com/@bananaguy80/the-great-filter-theory-are-we-a-filter-f73a2f1e933f https://medium.com/@bananaguy80/the-great-filter-theory-are-... We should ask why the newspapers silence this stuff. Why my congress men has an unlisted email, his contact form is broken, doesnt answer emails, facebook etc. And then get guns for your bull E.T.s.

omginternets 11y ago

I actually have an answer for this: Stop talking about "psychology" like it's a monolithic thing. Start by separating "hard" psychology (perception, attention, psychophysics, etc) from social psychology. Papers about saccadic adaptation are, I suspect, much more reproducible than those about implicit association.

simonster 11y ago
The OSC Science paper separates cognitive and social psychology papers and journals, and cognitive psychology papers replicate at about twice the rate of social psychology papers. But still only a bit more than half of the cognitive papers replicate, so I think there are still deeper issues.
- hga 11y ago
  But still only a bit more than half of the cognitive papers replicate, so I think there are still deeper issues. That puts them in the general level of quality for all bio-medical research, so, yeah, deeper issues indeed, but I find it fascinating that this sub-group is so good.
  - omginternets 11y ago
    This because that sub-group is very, very close to medical research. The problem with social psychology, on the whole, is that it's highly politicized. This is especially true with regards to any study involving race, gender, or "identity" (in the sense of identity politics).
Lawtonfogle 11y ago
One thing I noticed is that the more focused the subset of psychology, the better the results. For example psychology that looks at a group of neurons and how they function is vastly better than psychology looking at how some early life event X leads to later problems in life.
- 11y ago

analog31 11y ago

One small suggestion: Eliminate "published and outa here," the policy where students have to publish their MS and Phd work in order to receive those degrees. All that does is to fill the journals with crap.

omginternets 11y ago
I agree in principle but oftentimes the research done by grad-students is part of a project that is spearheaded by a career researcher. I question the notion that MS and PhD work is filling journals with crap. Instead, I think there are three problems: 1. The inherent problem of inference. When you set p < .05, then 5% of studies will yield false positives. This can be mitigated by a variety of statistical approaches, but it's an irreducible problem on a fundamental level. 2. Certain (sub)disciplines of science aren't very scientific. In particular, they suffer from a high incidence of unfalsifiable claims and hand-wavy definitions. A good example, I think, is the shame vs guilt literature. 3. Certain (sub)disciplines of science are highly politicized. I point my finger at anything involving "identity" (in the sense of "identity politics"), "diversity" (sex, age, ethnicity, sexual orientation, etc) or "bias". Regarding point 3, take a look at the faculty and students of any university you'd like and take stock of how many Black people are working on racism, women on sexism, LGBT on sexual orientation, etc. As a cognitive neuroscientist, I don't want to claim that people can't study themselves, but in light of the serious reproducibility issues that social psychology is facing, I think this correlation is anything but trivial. I strongly suspect that the "hard psych vs social psych" dichotomy can be further broken down, and that the sub-disciplines listed in point 3 account for a disproportionate quantity of false-positive results, on accounts of flawed methodology. I suspect that much of this research is actually an attempt to validate political opinion. I may be wrong about all of this, but at least the core claim testable with a handful of linear regressions.
- analog31 11y ago
  All good stuff. Thanks. I'm just saying, don't make publication a requirement for graduation. I'm not a statistician, but as I understand it, a problem with the p value is that the standard calculation hinges on assumptions that are not necessarily met by the data, and if those assumptions were correctly accounted for, p would be higher. If p < 0.05, and in practice 50% of published results are bunk, then regardless of the underlying math, a practical rule of thumb seems to be: Multiply p by 10.

medymed 11y ago

Redistribute most of the $5 billion dollars of psychology and behavioral-related NIH research funding to historically high ROI areas like infectious disease research. http://report.nih.gov/categorical_spending.aspx http://report.nih.gov/categorical_spending.aspx

DennisP 11y ago

A friend of mine is a social psychology researcher, who's published work destroying the statistics in other papers. He says this claim of unreliability is a lot more dubious than the media portrays. In many cases, they say a finding was not reproduced simply because the new study didn't quite meet the cutoff for "significance." However, if you do a meta-analysis of both studies, you find that the likelihood of the effect being real is higher, rather than lower. In other cases, the replications were done poorly. The first step of many studies is calibration. You do tests to find out what sort of things your subjects are familiar with, and calibrate questions accordingly. A common mistake in the replications was to skip the calibration step, and simply reuse the questions on subjects with different backgrounds.

Asbostos 11y ago
So we need the original authors to come back with their own re-replication. Instead of throwing their published paper over the wall and moving on, they are the people in the best position to actually defend it and address problems with replications. This publish-and-forget behavior annoys me about researchers. Sometimes I find results that look like they'll useful for my (non-academic) work, but the paper containing them is the final say on the matter. There's no version 2, there's no other work improving on it. It's just a dead end.

subliminalzen 11y ago

It's not just psychological research that is deeply flawed. Only a quarter of scientific drug research is successfully reproduced as well.[1] Carl Jung claimed one of the chief factors responsible for mass brainwashing is scientific rationality.[2] Society worships the Goddess of Reason while frowning down on "irrational" and non-verifiable religious testimony. Now that science is proven to be systemically corrupt, what will "rational" people base their understanding on? Aside: I designed a personality test / psychoanalytical tool that was inspired by Carl Jung. It's called Critical Stimulus and it can be found at https://www.thegamecrafter.com/games/critical-stimulus https://www.thegamecrafter.com/games/critical-stimulus. The printable version can be downloaded at gumroad: https://gumroad.com/l/criticalstimulus https://gumroad.com/l/criticalstimulus . [1] http://www.economist.com/news/briefing/21588057-scientists-think-science-self-correcting-alarming-degree-it-not-trouble http://www.economist.com/news/briefing/21588057-scientists-t... [2] https://en.wikipedia.org/wiki/Self_in_Jungian_psychology https://en.wikipedia.org/wiki/Self_in_Jungian_psychology

omginternets 11y ago
>Now that science is proven to be systemically corrupt, what will "rational" people base their understanding on? This is not a new notion in the least. The "rational" people will continue doing what they've always done: revising their conclusions. Science is a process, and while we can question the notion of "scientific progress" in philosophical terms, this is neither a new idea nor evidence that science doesn't work. It's certainly not evidence that science is no better than irrational thinking. >Carl Jung claimed one of the chief factors responsible for mass brainwashing is scientific rationality. Few people take psychoanalysts seriously these days, in large part because of their long record of absurd claims and shoddy clinical work. Jungian theory has it's place in a conversation about literary theory, but not in a conversation about science.
- deleted 11y ago
  [deleted]
- drumdance 11y ago
  I don't know about psychoanalysis as defined by Freud and Jung, but therapy in general is an intensely personal experience. I'm sure there are scientific aspects that can be brought to bear on a situation, but for a lot of people the "literary theory" part is just as helpful. Especially when you stop to think how much neurosis is fueled by pop culture (i.e. status envy).

rjruxhchd 11y ago

Far more worrying is that only a third of economics findings can be replicated without the author's help, and yet these are the supposedly scientific findings our politicians insist on basing policy off of. Even if policy seems counterintuitive and clearly harmful to the general public, we're given the explanation "because economics says so." It's clear now from the Federal Reserve replicability study that we're being duped. http://www.federalreserve.gov/econresdata/feds/2015/files/2015083pap.pdf http://www.federalreserve.gov/econresdata/feds/2015/files/20...

eru 11y ago
And if I read that right, they were just trying to run the code provided with the papers again.
chongli 11y ago
Economists are the modern equivalent to the oracles at Delphi. They speak in gibberish and the priesthood (politicians) make the interpretations which coincidentally happen to favour themselves and their friends.
- deleted 11y ago
  [deleted]
- sopooneo 11y ago
  I feel there's truth to that, but most at the edges. At core there is stuff economists tend to very widely agree on. There was a Planet Money episode outlining some pretty drastic policy changes that a panel of economists across the political spectrum all stood behind. Here's a write up: http://www.npr.org/sections/money/2012/07/19/157047211/six-policies-economists-love-and-politicians-hate http://www.npr.org/sections/money/2012/07/19/157047211/six-p...

geomark 11y ago

Let's be clear on the differencen between reproducible and replication. Reproducible is when an experiment's data and methods are published and other researchers can reproduce the same statistical results. An experiment that has been replicated is a new experiment that applies the same methods to a new set of subjects and arrives at statistically equivalent results. The distinction is important. Reproducible research is a lot easier but there are still famous examples of attempts to reproduce published results that revealed serious flaws. Cases where the data are kept secret to thwart reproducibility are thoroughly suspect. Replication is a harder problem, both in being able to perform a potentially costly and difficult experiment, and in the statistical analysis to show equivalent results.

nonbel 11y ago
>"Let's be clear on the difference between reproducible and replication." I did not find this clear. What is the distinction you are trying to make?
- geomark 11y ago
  Reproduce: take the published data and methods, rerun the analysis, see if you get the same results. No new experiment. Replicate: perform a new experiment with new subjects, apply the same methods, see if you get statistically equivalent results.
  - nonbel 11y ago
    Thanks, I had not heard that distinction in terms before. I suppose it would be good to have, but I would not assume people are using them in that way.
    - geomark 11y ago
      Some people are making a lot of noise about making the distinction, like the guys teaching data science at John Hopkins. It's because both are important but they are distinctly different activities.

nonbel 11y ago

Now what? You need to attempt independent replications of every published claim going forward. It is clear a single published result is unreliable. This is well known but apparently needed to be rediscovered by those who misunderstand the meaning of a p-value. >"The first is a p-value, which estimates the probability that the result was arrived at purely by chance and is a false positive. (Technically, the p-value is the chance that the result, or a stronger result, would have occurred even when there was no real effect.) Generally, if a statistical test shows that the p-value is lower than 5%, the study’s results are considered “significant” – most likely due to actual effects." No, there is endless literature on this. Just to start: http://andrewgelman.com/2015/07/21/a-bad-definition-of-statistical-significance-from-the-u-s-department-of-health-and-human-services-effective-health-care-program/ http://andrewgelman.com/2015/07/21/a-bad-definition-of-stati...

kyllo 11y ago
You need to attempt independent replications of every published claim going forward. Taking this opportunity to plug an open source application that I work on, for cataloging and verifying independent replications, called Curate Science: https://www.curatescience.org/ https://www.curatescience.org/
geomark 11y ago
More to the point is how "replication" is defined. This quote covers it: "Note that the 36% figure comes from a definition of replication that mimics the definition used by regulatory agencies: results are considered replicated if a p-value < 0.05 was reached in both the original study and the replicated one." If you read the post where that quote comes from [1] they make a number of points about how a better and more rigorous definition of replicated is needed, because p-value alone doesn't tell the whole story. [1] http://simplystatistics.org/2015/10/20/we-need-a-statistically-rigorous-and-scientifically-meaningful-definition-of-replication/ http://simplystatistics.org/2015/10/20/we-need-a-statistical...
- DennisP 11y ago
  So say one study has a p-value of .04, meaning there's a .04 probability that there's no effect and the results occurred by chance. A replication comes along and gets a p-value of .06, so it gets counted as a failed replication. And yet, the probability that both results happened by chance is only .0024.

matthferguson 11y ago

now what ... Accurately label psychology and the social sciences as philosophy not science.

millermp12 11y ago

When the science goes against the Narrative, do what our church fathers did: go with Scripture. First, the American Anthropological Association, next the APA? They might as well come of the closet and stop pretending so I can stop putting dick quotes around social "scientists." https://www.insidehighered.com/news/2010/11/30/anthroscience https://www.insidehighered.com/news/2010/11/30/anthroscience

themetrician 11y ago

The reliability of Psychology throughout the 20th century is covered in depth in the book The Culture of Critique, where the exact persons responsible for fostering in Psychology an anti-Science environment are discussed.

gregn 11y ago

From 10 years of working as a sysadmin in a Psych department, one constant in life I've learned is that Psychologists are the most incompetent, fanciful egoists in all the sciences. They have no idea what is going on basically. Or, put as a friend of mine said, "People who go into Psychology are precisely those that do not intuitively understand it already." Implying that most of us have a decent, commonsense appraisal of how things work, but psych people are the most clueless of the clueless, that's why they go into a field (first falling for its false promises) that professes to explain life's mysteries to them. None of this is surprising, except perhaps that they scored so far away from the median.

MollyR 11y ago

This is quite frightening considering how people try to set government policies based on them.