4 ms·

Approx. last week there was a big study that investigated whether other studies replicate or not. Here's a fun quiz where people can guess the results: https:/

by oerpli 8y ago

Approx. last week there was a big study that investigated whether other studies replicate or not. Here's a fun quiz where people can guess the results: https://80000hours.org/psychology-replication-quiz/ https://80000hours.org/psychology-replication-quiz/ One of the trends was that priming studies usually failed. I would assume that the quality of study form OP is similar.

trishume 8y ago
Remembering exactly this I went and found the paper and looked and the sample sizes aren't that small, I couldn't see any missing comparisons, and the relevant P values are less than 0.01 so not obviously super P-hacked. On the other hand it is a priming study that would probably not have been published if the result went the other way, and it's a newsworthy result. All indicators of non-replication. I got 88% accuracy on the replication quiz and if this was on the quiz I would have marked it as unsure. I don't think it's obviously bogus, but also I'm not totally convinced.
- JumpCrisscross 8y ago
  > it is a priming study Wasn’t the original study which claimed to “discover” priming recently debunked as not replicable?
mockingbirdy 8y ago
Would love to play this, but unfortunately: > This page is not set up to run this program.
smudgymcscmudge 8y ago
I did that quiz. Part way through I figured out to always choose no for priming and yes for anything related to game theory.
noobermin 8y ago
Allow me to hijack this thread. There is something very similar to the issue around repeatability of results in Psych fields and the lack of experimental evidence in theoretical particle physics which has been the focus of previous articles on HN. Both fields are dealing with large elephants in their respective fields in which some people know what the issue is, but rather than having a widespread reckoning, people keep publishing papers as if everything is okay. If anything, this demonstrates a problem that peer-reviewed science is not able to fix. I have my own thoughts, it's due to the siloed nature of science today, given publish of perish forces us to keep our head down and keep publishing whilst ignoring the big picture problems with the hope someone with more funding in a more prestiguous position will do something. It's more problematic in Psych though, because people and their quality of life are at stake which is not a situation in theoretical physics.

by oerpli 8y ago

trishume 8y ago

Remembering exactly this I went and found the paper and looked and the sample sizes aren't that small, I couldn't see any missing comparisons, and the relevant P values are less than 0.01 so not obviously super P-hacked. On the other hand it is a priming study that would probably not have been published if the result went the other way, and it's a newsworthy result. All indicators of non-replication. I got 88% accuracy on the replication quiz and if this was on the quiz I would have marked it as unsure. I don't think it's obviously bogus, but also I'm not totally convinced.

JumpCrisscross 8y ago
> it is a priming study Wasn’t the original study which claimed to “discover” priming recently debunked as not replicable?

mockingbirdy 8y ago

Would love to play this, but unfortunately: > This page is not set up to run this program.

smudgymcscmudge 8y ago

I did that quiz. Part way through I figured out to always choose no for priming and yes for anything related to game theory.

noobermin 8y ago

Allow me to hijack this thread. There is something very similar to the issue around repeatability of results in Psych fields and the lack of experimental evidence in theoretical particle physics which has been the focus of previous articles on HN. Both fields are dealing with large elephants in their respective fields in which some people know what the issue is, but rather than having a widespread reckoning, people keep publishing papers as if everything is okay. If anything, this demonstrates a problem that peer-reviewed science is not able to fix. I have my own thoughts, it's due to the siloed nature of science today, given publish of perish forces us to keep our head down and keep publishing whilst ignoring the big picture problems with the hope someone with more funding in a more prestiguous position will do something. It's more problematic in Psych though, because people and their quality of life are at stake which is not a situation in theoretical physics.