5 ms·

There's also the reproduction crisis[1], if you weren't aware of it already. When your field can only have less than half of its top papers reproduced in full,

There's also the reproduction crisis[1], if you weren't aware of it already. When your field can only have less than half of its top papers reproduced in full, something is very wrong with your field. It's to the point where one can safely assume that any psych paper (but especially social psych) is garbage until such time as it's been replicated a time or two. [1]: https://www.nature.com/news/over-half-of-psychology-studies-fail-reproducibility-test-1.18248 https://www.nature.com/news/over-half-of-psychology-studies-...

Tycho 8y ago
And consider this: in psychology we actually have the possibility of running and repeating controlled experiments. Think how bad things must be in other fields where that’s not the case.
feanaro 8y ago
> It's to the point where one can safely assume that any psych paper (but especially social psych) is garbage until such time as it's been replicated a time or two. Good, because it is probable that what we should assume this for any science[1]. [1]: http://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124 http://journals.plos.org/plosmedicine/article?id=10.1371/jou...
- neuromantik8086 8y ago
  Everyone assumes that science is filled with neat, well-manicured lab notebooks, pristine methodology and fastidious characters. In reality, the current scientific culture is one of rampant distractions, where pretty much everyone is running around like a beheaded chicken. Woe unto those that might point out that a prominent researcher with tenure isn't using sound methods. That, as well as a generally toxic culture that makes Wall Street look good, was part of why I left. There have been some efforts towards creating a better culture (in particular, I've had my eye on Popper [1] and the "SciOps" movement and Software Carpentry [2], as well as various open access efforts). At the end of the day though, scarcity in terms of funding plus lack of job security plus medieval power dynamics are holding any substantial cultural shift back. [1] http://falsifiable.us/ http://falsifiable.us/ [2] https://software-carpentry.org/ https://software-carpentry.org/
c3534l 8y ago
Not only is the replication crisis not solely about psychology, the numbers are far worse than half for fields like chemistry, biology, and physics[0]. [0] https://en.wikipedia.org/wiki/Replication_crisis#Overall https://en.wikipedia.org/wiki/Replication_crisis#Overall
- ummonk 8y ago
  That's not what the numbers there are saying though. They're saying that over half of scientists have failed to replicate a single study (either their own or someone else's). It says nothing about the replicability of most studies.
- TangoTrotFox 8y ago
  This is completely incorrect, and that Wiki page has been written to be highly misleading to lead people to the same mistaken comparison as they're comparing entirely different numbers that look directly comparable. What that says is that e.g. 69% of scientists working in physics and engineering had failed to replicate some experiment at least once in their career. In other words if they performed 1000 replication attempts of other scientists' experiments and failed to reproduce the same results once, they would be counted in that survey. By contrast in psychology, one hallmark experiment tried to replicate the findings of 100 key studies in psychology from highly regarded psychology journals. And it was unable to replicate the findings of 64% of the studies. What's that number for physics? No idea other than that it's going to be much lower. Make sense? Wiki is, almost as a rule, just completely awful for contemporary or potentially controversial topics and this is a textbook example of that.
rusk 8y ago
The reproduction crisis affects all social sciences, and medicine to a certain degree. In fact anything where it's difficult to isolate and control for the phenomena are facing this. Climate science, Economics and food science are all suffering similar issues. I think this speaks more to the nature of academia, and the commercialisation of publication than the validity of the field in and of itself.
UncleMeat 8y ago
At least psych is addressing the reproduction crisis. Other fields are just ignoring it (cough, ML).

Tycho 8y ago

And consider this: in psychology we actually have the possibility of running and repeating controlled experiments. Think how bad things must be in other fields where that’s not the case.

feanaro 8y ago

> It's to the point where one can safely assume that any psych paper (but especially social psych) is garbage until such time as it's been replicated a time or two. Good, because it is probable that what we should assume this for any science[1]. [1]: http://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124 http://journals.plos.org/plosmedicine/article?id=10.1371/jou...

neuromantik8086 8y ago
Everyone assumes that science is filled with neat, well-manicured lab notebooks, pristine methodology and fastidious characters. In reality, the current scientific culture is one of rampant distractions, where pretty much everyone is running around like a beheaded chicken. Woe unto those that might point out that a prominent researcher with tenure isn't using sound methods. That, as well as a generally toxic culture that makes Wall Street look good, was part of why I left. There have been some efforts towards creating a better culture (in particular, I've had my eye on Popper [1] and the "SciOps" movement and Software Carpentry [2], as well as various open access efforts). At the end of the day though, scarcity in terms of funding plus lack of job security plus medieval power dynamics are holding any substantial cultural shift back. [1] http://falsifiable.us/ http://falsifiable.us/ [2] https://software-carpentry.org/ https://software-carpentry.org/

c3534l 8y ago

Not only is the replication crisis not solely about psychology, the numbers are far worse than half for fields like chemistry, biology, and physics[0]. [0] https://en.wikipedia.org/wiki/Replication_crisis#Overall https://en.wikipedia.org/wiki/Replication_crisis#Overall

ummonk 8y ago
That's not what the numbers there are saying though. They're saying that over half of scientists have failed to replicate a single study (either their own or someone else's). It says nothing about the replicability of most studies.
TangoTrotFox 8y ago
This is completely incorrect, and that Wiki page has been written to be highly misleading to lead people to the same mistaken comparison as they're comparing entirely different numbers that look directly comparable. What that says is that e.g. 69% of scientists working in physics and engineering had failed to replicate some experiment at least once in their career. In other words if they performed 1000 replication attempts of other scientists' experiments and failed to reproduce the same results once, they would be counted in that survey. By contrast in psychology, one hallmark experiment tried to replicate the findings of 100 key studies in psychology from highly regarded psychology journals. And it was unable to replicate the findings of 64% of the studies. What's that number for physics? No idea other than that it's going to be much lower. Make sense? Wiki is, almost as a rule, just completely awful for contemporary or potentially controversial topics and this is a textbook example of that.

rusk 8y ago

The reproduction crisis affects all social sciences, and medicine to a certain degree. In fact anything where it's difficult to isolate and control for the phenomena are facing this. Climate science, Economics and food science are all suffering similar issues. I think this speaks more to the nature of academia, and the commercialisation of publication than the validity of the field in and of itself.

UncleMeat 8y ago

At least psych is addressing the reproduction crisis. Other fields are just ignoring it (cough, ML).