4 ms·

Reproducibility in social psychology is less than 50% [1]. So yes I do think the majority of psych researchers are making stuff up (through p-hacking and other

Reproducibility in social psychology is less than 50% [1]. So yes I do think the majority of psych researchers are making stuff up (through p-hacking and other nefarious means) in order to get funding and tenure 1. https://replicationindex.com/2018/11/20/how-replicable-is-psychological-science/ https://replicationindex.com/2018/11/20/how-replicable-is-ps...

amirkdv 5y ago
You may be assigning more intent than is necessary to explain the (very real) reproducibility problem in psych. Not to belabor this but to note the irony here: there is probably a similar psychological (ehem) phenomenon that happens to a psyc grad student who lets their presumptions bleed into their interpretation of a measurement. > The first principle is that you must not fool yourself — and you are the easiest person to fool. R. Feynman
CharlesW 5y ago
> Reproducibility in social psychology is less than 50%. That doesn't seem particularly terrible in comparison to scientific reproducibility in other fields. What am I missing? https://en.wikipedia.org/wiki/Replication_crisis https://en.wikipedia.org/wiki/Replication_crisis
- temp8964 5y ago
  People usually underestimate how difficult it is to do experiments, especially when you want to get quantitatively consistent results. Physics experiments sound easier than human experiments, but if you really do it and be serious with the numbers, you will find even high school physics experiments are really hard. For example, try to confirm F=ma, or measure local gravity to the third digit.
freshair 5y ago
They could be fooling themselves too, not knowingly spreading falsehoods. A lot of of what we call p-hacking can happen by accident by somebody who isn't very good at statistics or just isn't being careful.
ampdepolymerase 5y ago
We do have replicable, quantifiable, social psychology. Facebook and Cambridge Analytica are infamous for that. Unfortunately whenever experiments are carried out, you get end up with an uproar over issues of privacy, accountability, and informed consent.
- loopz 5y ago
  Spying and manipulation are not necessarily reproducible hypotheses. In case of FB and CA, external conditions changed.
temp8964 5y ago
50% is reproducibility of experimental studies. There are fields don't even do experiment. There are fields don't even run statistics. There are fields don't even use logic.....
fastaguy88 5y ago
A question the 50% figure raises is - what is the reproducibility rate for “real” science? This is difficult to estimate. Considering the dramatic improvements in cancer therapy, one might argue that cancer research is a “real” science, but there are reproducibility issues in that field as well. [https://www.nature.com/news/cancer-reproducibility-project-releases-first-results-1.21304 https://www.nature.com/news/cancer-reproducibility-project-r...] And the link to the replication index site connects us to an article about John Iaonnidis, who suggests that most medical research results are wrong. (The article concludes that perhaps only 27% are false discoveries.) So what is the threshold? As others have pointed out, it’s probably more about process than reproducibility, though reproducible results are certainly preferred.
- thu2111 5y ago
  I think that would depend on how much intersection there is between the sort of scientists doing non-replicable research (mostly in academia, it seems) and the sorts of scientists developing working cancer therapies (mostly in pharma?). I don't know if arguments about "real" science are especially useful. The nature of science has changed a lot of over time and will presumably keep changing. It takes us into questions of whether fields that can't do experiments are "real" science, for example. It also assumes replicability is the only way science can't be "real" but there are some fields (e.g. COVID research!) where papers are usually replicable but that is meaningless because the methodology is unsound in other ways, i.e. you'd just be replicating their mistakes. The biggest problems driving non-replicable research are the fact that so much "science" is now really just data analysis, and finding some spurious correlation that just about nudges over some arbitrary and very weak threshold for significance can be considered "science", even without anyone doing any kind of double check or followup. No real world experiment needs to be done at any point. And in some fields the connection to reality has been completely lost - public health is filled with papers that make predictions about the future trends in public data, and nobody ever follows up with papers that compute the correlation between the predictions and measured values. It would be easy, but they generally don't do it.
  - fastaguy88 5y ago
UncleMeat 5y ago
And they are basically the only field funding replication studies. That makes them more rigorous than many alternatives.

amirkdv 5y ago

You may be assigning more intent than is necessary to explain the (very real) reproducibility problem in psych. Not to belabor this but to note the irony here: there is probably a similar psychological (ehem) phenomenon that happens to a psyc grad student who lets their presumptions bleed into their interpretation of a measurement. > The first principle is that you must not fool yourself — and you are the easiest person to fool. R. Feynman

CharlesW 5y ago

> Reproducibility in social psychology is less than 50%. That doesn't seem particularly terrible in comparison to scientific reproducibility in other fields. What am I missing? https://en.wikipedia.org/wiki/Replication_crisis https://en.wikipedia.org/wiki/Replication_crisis

temp8964 5y ago
People usually underestimate how difficult it is to do experiments, especially when you want to get quantitatively consistent results. Physics experiments sound easier than human experiments, but if you really do it and be serious with the numbers, you will find even high school physics experiments are really hard. For example, try to confirm F=ma, or measure local gravity to the third digit.

freshair 5y ago

They could be fooling themselves too, not knowingly spreading falsehoods. A lot of of what we call p-hacking can happen by accident by somebody who isn't very good at statistics or just isn't being careful.

ampdepolymerase 5y ago

We do have replicable, quantifiable, social psychology. Facebook and Cambridge Analytica are infamous for that. Unfortunately whenever experiments are carried out, you get end up with an uproar over issues of privacy, accountability, and informed consent.

loopz 5y ago
Spying and manipulation are not necessarily reproducible hypotheses. In case of FB and CA, external conditions changed.

temp8964 5y ago

50% is reproducibility of experimental studies. There are fields don't even do experiment. There are fields don't even run statistics. There are fields don't even use logic.....

fastaguy88 5y ago

A question the 50% figure raises is - what is the reproducibility rate for “real” science? This is difficult to estimate. Considering the dramatic improvements in cancer therapy, one might argue that cancer research is a “real” science, but there are reproducibility issues in that field as well. [https://www.nature.com/news/cancer-reproducibility-project-releases-first-results-1.21304 https://www.nature.com/news/cancer-reproducibility-project-r...] And the link to the replication index site connects us to an article about John Iaonnidis, who suggests that most medical research results are wrong. (The article concludes that perhaps only 27% are false discoveries.) So what is the threshold? As others have pointed out, it’s probably more about process than reproducibility, though reproducible results are certainly preferred.

thu2111 5y ago
I think that would depend on how much intersection there is between the sort of scientists doing non-replicable research (mostly in academia, it seems) and the sorts of scientists developing working cancer therapies (mostly in pharma?). I don't know if arguments about "real" science are especially useful. The nature of science has changed a lot of over time and will presumably keep changing. It takes us into questions of whether fields that can't do experiments are "real" science, for example. It also assumes replicability is the only way science can't be "real" but there are some fields (e.g. COVID research!) where papers are usually replicable but that is meaningless because the methodology is unsound in other ways, i.e. you'd just be replicating their mistakes. The biggest problems driving non-replicable research are the fact that so much "science" is now really just data analysis, and finding some spurious correlation that just about nudges over some arbitrary and very weak threshold for significance can be considered "science", even without anyone doing any kind of double check or followup. No real world experiment needs to be done at any point. And in some fields the connection to reality has been completely lost - public health is filled with papers that make predictions about the future trends in public data, and nobody ever follows up with papers that compute the correlation between the predictions and measured values. It would be easy, but they generally don't do it.
- fastaguy88 5y ago

UncleMeat 5y ago

And they are basically the only field funding replication studies. That makes them more rigorous than many alternatives.