11 ms·

The foot soldiers behind psychology’s replication crisis

98 points by sampo 8y ago

ISL 8y ago
Those attempting to make a science more reproducible are not behind the crisis, they are part of the solution.
- bassman9000 8y ago
  "It’s a hard enough life to be a scientist," she says. "If we want our best and brightest to be scientists, this is not the way to do it." And he’s a skeptic of this new generation of skeptics. For starters, Nisbett doesn’t think direct replications are efficient or sensible; instead he favors so-called conceptual replication, which is more or less taking someone else’s interesting result and putting your own spin on it. It's gotta be about feelings, then.
  - nonbel 8y ago
    Yep, his idea is for psychologists to never actually double check each other's work. Hes probably scared of what would be found. Edit: And of course "conceptual replications" should also be done. Its just that they are not a replacement for direct replications.
    - mcguire 8y ago
      How often are direct replications done in other sciences?
      - nonbel 8y ago
        Direct replications are done in science 100% of the time something becomes fact. Eg, see my other post here: https://news.ycombinator.com/item?id=17981869 https://news.ycombinator.com/item?id=17981869 Btw, I dont consider either psychology nor medical research to be a science at this point. I just call them "research", which seems neutral enough to me.
peoplewrong 8y ago
disagreement and people being wildly wildly wrong in good faith [and other people pointing it out!] is an essential part of the scientific process. it is difficult because academics often closely associate themselves with the status of their scientific contributions. No one really likes killing your heroes but sometimes it must be done. If psychology claims to be a science it MUST collectively accept this. However we must be sure to deal kindly with the people behind the ideas.
- candiodari 8y ago
  > disagreement and people being wildly wildly wrong in good faith Unfortunately but the most important basic studies of psychiatry were bullshit. And maybe they were in good faith, but it's incredibly hard to believe that. Freud, for instance, was either serial sex offender and mass-accomplice to sexual abuse, or an incredible rube ... You see Freud saw sexual fantasies everywhere in women. They would fantasize about getting raped by this or that family member. There's a tiny little problem with that ... It's come out in more than a few cases that these patients were in fact raped, assaulted and ... by family members and others. Afterwards these "patients" saw everyone's intentions to be sexual and expressed that by seeing sexual abuse everywhere and worrying about what would happen to them. Strange, isn't it ? Freud never once saw it. God knows how many women tried to convince him to help them get justice, but it must have been over 100 at least. Never once did he help. Maybe he knew, maybe he didn't. He did lock quite a few of them away. https://en.wikipedia.org/wiki/The_Freudian_Coverup https://en.wikipedia.org/wiki/The_Freudian_Coverup But no worries ! Since then we've vastly improved efficiency: https://en.wikipedia.org/wiki/Political_abuse_of_psychiatry https://en.wikipedia.org/wiki/Political_abuse_of_psychiatry
  - Retric 8y ago
    It's generally accepted that Science increases accuracy over time, but I don't think people understand what that looks like. Intuitively people tend to think of random ideas to be 50/50 true or false. But, the space of possible vs actual means random ideas are almost always false. So, fields like medicine demonstrate you can spend generations looking into something and still be wildly wrong. Freud like all early researchers had no idea what was going on. But, as a baseline to be improved upon he still provides part of a useful foundation for the discipline.
morley 8y ago
Is there a list of well-known studies that have been overturned because they haven't been reproducible? I'm wondering how much "common knowledge" that has come from those studies is actually true or not.
- nonbel 8y ago
  There are, but the "replication studies" tend to change little details for whatever reason so they aren't really checking for reproduciblity anyway. I don't mean details like where/when the study was performed, because obviously the original authors thought it would generalize beyond their exact sample. I mean they will change the methods to ask different questions on a survey, etc. Also, they use statistical significance vs not to determine if the same result was observed (rather than having quantitatively similar results)... The entire situation is ridiculous. You can start here though: https://en.wikipedia.org/wiki/Reproducibility_Project https://en.wikipedia.org/wiki/Reproducibility_Project
- nnash 8y ago
  Science, in general, has had a reproducibility problem for some time now. There was a big article about it in Nature a couple of years ago. https://www.nature.com/news/1-500-scientists-lift-the-lid-on-reproducibility-1.19970 https://www.nature.com/news/1-500-scientists-lift-the-lid-on...
iron0013 8y ago
This is not pyschology's replication crisis. Every field has replication problems, psychology is just one of the only fields taking it seriously.
- acover 8y ago
  Source? I don't hear about rampant replication problems in chemistry/physics. https://en.wikipedia.org/wiki/Replication_crisis https://en.wikipedia.org/wiki/Replication_crisis
  - nordsieck 8y ago
    This[0] is a great video talking about problems with lots of studies in biology-adjacent fields including, but not limited to, psychology and nutrition. [0] John Ioannidis: "Reproducible Research: True or False?" https://www.youtube.com/watch?v=GPYzY9I78CI https://www.youtube.com/watch?v=GPYzY9I78CI
    - acover 8y ago
      Interesting! Sadly medicine has a replication crisis. I don't think chemistry or physics does.
  - mcguire 8y ago
    I'll just leave a mention of the current state of theoretical physics here: https://www.nature.com/articles/d41586-018-05374-9 https://www.nature.com/articles/d41586-018-05374-9
    - tormeh 8y ago
      That's a criticism of a bunch of heuristics we've had for selecting promising theories. No popular theory has been proven wrong. It's just that we've been stuck for a really long time. And we need some kind of heuristics, otherwise we're just going to overfit to our data.
  - schuetze 8y ago
acover 8y ago
Ranking of universities by replicability https://replicationindex.wordpress.com/2016/03/02/replicability-ranking-of-psychology-departments/ https://replicationindex.wordpress.com/2016/03/02/replicabil...
bnchrch 8y ago
There's a lot of discussion here on why Psychology has this problem more so than fields outside the social science realm. I think we're all missing what is fundamentally flawed about academic psychology; and it's not their methodology. In North America (perhaps elsewhere) you are required to have at least a Master's degree to practise Psychology and you should have a doctorate if you want any mobility with your practise. This leads people who have no interest in academia having to find a way to convince people they've discovered something new and novel so that they can go apply what has already been discovered. It's no surprise many of this studies can't be replicated! They were designed from the beginning to lead to a significant findings so that someone could write their dissertation, get their doctorate and get the hell out of there. And you know what? I do not blame them in the slightest. Academia is a nightmare and it's holding a whole profession hostage.
- rhizome 8y ago
  Heck, computer science has the same problem, and probably will as long as psychology, since like every person, every business is different. Replication difficulties follow naturally.
  - dickeytk 8y ago
    Academia is certainly not holding computer science hostage like this. Nowhere near as many people have graduate degrees in CS as people practicing their Psychology degrees.
    - harimau777 8y ago
      I think computer science is still held hostage to academia if you are interested in doing more creative work. There is a difference between "practical research" where you are developing new ways to solve problems or apply technology and "academic research" where you are developing computer science theory. However, if you want to do practical research, then opportunities are limited if you don't have a PhD. Without one it seems like the primary options are CRUD apps. It doesn't seem like this is the case in other engineering disciplines or even outside of engineering.
mcguire 8y ago
"One evening at the SIPS conference, after sessions had been concluded and beers consumed, a researcher asked Heathers for his opinion of a study that seemed suspicious. When Heathers decided he’d heard enough to render a verdict, he took a few steps back and began to shout. ""Give me a B!" ""B!" the assembled scientists replied. ""Give me a U!"" And here we see how more bullshit gets started. Deciding that something sounds bad means it is wrong is exactly the same error as deciding that something sounds good means it is right.
- jcranmer 8y ago
  > Deciding that something sounds bad means it is wrong is exactly the same error as deciding that something sounds good means it is right. False equivalence. A correct paper inherently has no flaws, while a wrong paper inherently has at least one flaw. It is very possible to be able to immediately detect some classes of flaws, and that in no way means that one can immediately find all of the flaws. When I was a TA, I found several people who cheated. One person I discovered because there's no way a struggling student in an introductory C course is going to pull out code with gotos and register keyword usage plastered everywhere. Do I claim to have caught everyone who cheated? Hell no.
  - eslaught 8y ago
    I don't know if you've ever been subject to scientific peer review or not. One of the most frustrating things is reviewers who misunderstand the paper, but are convinced they've found some sort of problem with it. Admittedly, these sorts of issues usually point to places where the paper could be more clear. However, the point still stands that it is far easier to misunderstand something, than it is to understand it and find a flaw. I would be very, very suspicious of any scientist claiming to have found a flaw in a work they've only been given a superficial description of. It may be accurate to say they have a hunch of where a flaw might be. But without really spending time with a paper it would foolish to claim any degree of certainty. Only very low quality papers can be rejected so quickly.
    - TangoTrotFox 8y ago
jancsika 8y ago
Are there examples of these "data thugs" causing quality scientific work to be unduly criticized? For example, good faith research that cannot be replicated is part of science. Are there researchers who have done such good faith research (no p-hacking, no egregious statistical errors) who now have to endure harassment and low-quality criticism because of the work of the "data thugs?"
- mmirate 8y ago
  "Bad faith" research is almost certainly non-replicable, but non-replicable research isn't necessarily in "bad faith". But that doesn't matter, because non-replicable research is still non-replicable research. If you study a nonextant phenomenon enough times, then you can produce a null-hypothesis rejection anyway; and if you study a nonextent phenomenon once, then that one study has a chance (albeit minute) of rejecting the null hypothesis anyway. The first is bad faith, the second is either bad luck or incompetence, but neither is an accurate representation of whatever was being studied. > good faith research that cannot be replicated is part of science. To science's detriment, yes.
  - nonbel 8y ago
    >"But that doesn't matter, because non-replicable research is still non-replicable research" Exactly, the people who say direct replications are unnecessary are the ones fostering an environment for fraud. To the scientist it doesn't matter much why you couldn't describe what you did well enough for other people to repeat it. Maybe you made it up, maybe it depended upon the (unmeasured) magnetic field in your room, whatever.

98 points by sampo 8y ago

ISL 8y ago

Those attempting to make a science more reproducible are not behind the crisis, they are part of the solution.

bassman9000 8y ago
"It’s a hard enough life to be a scientist," she says. "If we want our best and brightest to be scientists, this is not the way to do it." And he’s a skeptic of this new generation of skeptics. For starters, Nisbett doesn’t think direct replications are efficient or sensible; instead he favors so-called conceptual replication, which is more or less taking someone else’s interesting result and putting your own spin on it. It's gotta be about feelings, then.
- nonbel 8y ago
  Yep, his idea is for psychologists to never actually double check each other's work. Hes probably scared of what would be found. Edit: And of course "conceptual replications" should also be done. Its just that they are not a replacement for direct replications.
  - mcguire 8y ago
    How often are direct replications done in other sciences?
    - nonbel 8y ago
      Direct replications are done in science 100% of the time something becomes fact. Eg, see my other post here: https://news.ycombinator.com/item?id=17981869 https://news.ycombinator.com/item?id=17981869 Btw, I dont consider either psychology nor medical research to be a science at this point. I just call them "research", which seems neutral enough to me.

peoplewrong 8y ago

disagreement and people being wildly wildly wrong in good faith [and other people pointing it out!] is an essential part of the scientific process. it is difficult because academics often closely associate themselves with the status of their scientific contributions. No one really likes killing your heroes but sometimes it must be done. If psychology claims to be a science it MUST collectively accept this. However we must be sure to deal kindly with the people behind the ideas.

candiodari 8y ago
> disagreement and people being wildly wildly wrong in good faith Unfortunately but the most important basic studies of psychiatry were bullshit. And maybe they were in good faith, but it's incredibly hard to believe that. Freud, for instance, was either serial sex offender and mass-accomplice to sexual abuse, or an incredible rube ... You see Freud saw sexual fantasies everywhere in women. They would fantasize about getting raped by this or that family member. There's a tiny little problem with that ... It's come out in more than a few cases that these patients were in fact raped, assaulted and ... by family members and others. Afterwards these "patients" saw everyone's intentions to be sexual and expressed that by seeing sexual abuse everywhere and worrying about what would happen to them. Strange, isn't it ? Freud never once saw it. God knows how many women tried to convince him to help them get justice, but it must have been over 100 at least. Never once did he help. Maybe he knew, maybe he didn't. He did lock quite a few of them away. https://en.wikipedia.org/wiki/The_Freudian_Coverup https://en.wikipedia.org/wiki/The_Freudian_Coverup But no worries ! Since then we've vastly improved efficiency: https://en.wikipedia.org/wiki/Political_abuse_of_psychiatry https://en.wikipedia.org/wiki/Political_abuse_of_psychiatry
- Retric 8y ago
  It's generally accepted that Science increases accuracy over time, but I don't think people understand what that looks like. Intuitively people tend to think of random ideas to be 50/50 true or false. But, the space of possible vs actual means random ideas are almost always false. So, fields like medicine demonstrate you can spend generations looking into something and still be wildly wrong. Freud like all early researchers had no idea what was going on. But, as a baseline to be improved upon he still provides part of a useful foundation for the discipline.

morley 8y ago

Is there a list of well-known studies that have been overturned because they haven't been reproducible? I'm wondering how much "common knowledge" that has come from those studies is actually true or not.

nonbel 8y ago
There are, but the "replication studies" tend to change little details for whatever reason so they aren't really checking for reproduciblity anyway. I don't mean details like where/when the study was performed, because obviously the original authors thought it would generalize beyond their exact sample. I mean they will change the methods to ask different questions on a survey, etc. Also, they use statistical significance vs not to determine if the same result was observed (rather than having quantitatively similar results)... The entire situation is ridiculous. You can start here though: https://en.wikipedia.org/wiki/Reproducibility_Project https://en.wikipedia.org/wiki/Reproducibility_Project
nnash 8y ago
Science, in general, has had a reproducibility problem for some time now. There was a big article about it in Nature a couple of years ago. https://www.nature.com/news/1-500-scientists-lift-the-lid-on-reproducibility-1.19970 https://www.nature.com/news/1-500-scientists-lift-the-lid-on...

iron0013 8y ago

This is not pyschology's replication crisis. Every field has replication problems, psychology is just one of the only fields taking it seriously.

acover 8y ago
Source? I don't hear about rampant replication problems in chemistry/physics. https://en.wikipedia.org/wiki/Replication_crisis https://en.wikipedia.org/wiki/Replication_crisis
- nordsieck 8y ago
  This[0] is a great video talking about problems with lots of studies in biology-adjacent fields including, but not limited to, psychology and nutrition. [0] John Ioannidis: "Reproducible Research: True or False?" https://www.youtube.com/watch?v=GPYzY9I78CI https://www.youtube.com/watch?v=GPYzY9I78CI
  - acover 8y ago
    Interesting! Sadly medicine has a replication crisis. I don't think chemistry or physics does.
- mcguire 8y ago
  I'll just leave a mention of the current state of theoretical physics here: https://www.nature.com/articles/d41586-018-05374-9 https://www.nature.com/articles/d41586-018-05374-9
  - tormeh 8y ago
    That's a criticism of a bunch of heuristics we've had for selecting promising theories. No popular theory has been proven wrong. It's just that we've been stuck for a really long time. And we need some kind of heuristics, otherwise we're just going to overfit to our data.
- schuetze 8y ago

acover 8y ago

Ranking of universities by replicability https://replicationindex.wordpress.com/2016/03/02/replicability-ranking-of-psychology-departments/ https://replicationindex.wordpress.com/2016/03/02/replicabil...

bnchrch 8y ago

There's a lot of discussion here on why Psychology has this problem more so than fields outside the social science realm. I think we're all missing what is fundamentally flawed about academic psychology; and it's not their methodology. In North America (perhaps elsewhere) you are required to have at least a Master's degree to practise Psychology and you should have a doctorate if you want any mobility with your practise. This leads people who have no interest in academia having to find a way to convince people they've discovered something new and novel so that they can go apply what has already been discovered. It's no surprise many of this studies can't be replicated! They were designed from the beginning to lead to a significant findings so that someone could write their dissertation, get their doctorate and get the hell out of there. And you know what? I do not blame them in the slightest. Academia is a nightmare and it's holding a whole profession hostage.

rhizome 8y ago
Heck, computer science has the same problem, and probably will as long as psychology, since like every person, every business is different. Replication difficulties follow naturally.
- dickeytk 8y ago
  Academia is certainly not holding computer science hostage like this. Nowhere near as many people have graduate degrees in CS as people practicing their Psychology degrees.
  - harimau777 8y ago
    I think computer science is still held hostage to academia if you are interested in doing more creative work. There is a difference between "practical research" where you are developing new ways to solve problems or apply technology and "academic research" where you are developing computer science theory. However, if you want to do practical research, then opportunities are limited if you don't have a PhD. Without one it seems like the primary options are CRUD apps. It doesn't seem like this is the case in other engineering disciplines or even outside of engineering.

mcguire 8y ago

"One evening at the SIPS conference, after sessions had been concluded and beers consumed, a researcher asked Heathers for his opinion of a study that seemed suspicious. When Heathers decided he’d heard enough to render a verdict, he took a few steps back and began to shout. ""Give me a B!" ""B!" the assembled scientists replied. ""Give me a U!"" And here we see how more bullshit gets started. Deciding that something sounds bad means it is wrong is exactly the same error as deciding that something sounds good means it is right.

jcranmer 8y ago
> Deciding that something sounds bad means it is wrong is exactly the same error as deciding that something sounds good means it is right. False equivalence. A correct paper inherently has no flaws, while a wrong paper inherently has at least one flaw. It is very possible to be able to immediately detect some classes of flaws, and that in no way means that one can immediately find all of the flaws. When I was a TA, I found several people who cheated. One person I discovered because there's no way a struggling student in an introductory C course is going to pull out code with gotos and register keyword usage plastered everywhere. Do I claim to have caught everyone who cheated? Hell no.
- eslaught 8y ago
  I don't know if you've ever been subject to scientific peer review or not. One of the most frustrating things is reviewers who misunderstand the paper, but are convinced they've found some sort of problem with it. Admittedly, these sorts of issues usually point to places where the paper could be more clear. However, the point still stands that it is far easier to misunderstand something, than it is to understand it and find a flaw. I would be very, very suspicious of any scientist claiming to have found a flaw in a work they've only been given a superficial description of. It may be accurate to say they have a hunch of where a flaw might be. But without really spending time with a paper it would foolish to claim any degree of certainty. Only very low quality papers can be rejected so quickly.
  - TangoTrotFox 8y ago

jancsika 8y ago

Are there examples of these "data thugs" causing quality scientific work to be unduly criticized? For example, good faith research that cannot be replicated is part of science. Are there researchers who have done such good faith research (no p-hacking, no egregious statistical errors) who now have to endure harassment and low-quality criticism because of the work of the "data thugs?"

mmirate 8y ago
"Bad faith" research is almost certainly non-replicable, but non-replicable research isn't necessarily in "bad faith". But that doesn't matter, because non-replicable research is still non-replicable research. If you study a nonextant phenomenon enough times, then you can produce a null-hypothesis rejection anyway; and if you study a nonextent phenomenon once, then that one study has a chance (albeit minute) of rejecting the null hypothesis anyway. The first is bad faith, the second is either bad luck or incompetence, but neither is an accurate representation of whatever was being studied. > good faith research that cannot be replicated is part of science. To science's detriment, yes.
- nonbel 8y ago
  >"But that doesn't matter, because non-replicable research is still non-replicable research" Exactly, the people who say direct replications are unnecessary are the ones fostering an environment for fraud. To the scientist it doesn't matter much why you couldn't describe what you did well enough for other people to repeat it. Maybe you made it up, maybe it depended upon the (unmeasured) magnetic field in your room, whatever.