4 ms·

Do you hold an oracle which tells you which studies will not replicate, or is there some particular trait of the study that provably doesn't replicate?

reedjosh 5y ago
It's not about an Oracle. It's that the mentioning of a topic prior to some conclusion generating observation has practically never generated replicable results. Still, these studies make news all the time because they're easy to p-hack into some interesting conclusion.
- h0l0cube 5y ago
  Reading the study it appears they went to some pains in not revealing their intentions: > The caregiver and child were seated across from one another at a table. The child completed an unrelated experiment with the researcher, while the caregiver completed the Scarcity or Control manipulation survey on a tablet. When the caregiver had finished, the researcher left the room under the guise of loading a second survey onto the iPad, leaving the caregiver and child alone with a toy the experimenter happened to offer as she left. A video camera and/or tape recorder recorded their interactions. The conclusion could be that being reminded or queried about hardships has a chilling effect
tqi 5y ago
People are good at predicting this: https://www.theatlantic.com/science/archive/2018/08/scientists-can-collectively-sense-which-psychology-studies-are-weak/568630/ https://www.theatlantic.com/science/archive/2018/08/scientis...
- h0l0cube 5y ago
  > People To be specific: > They recruited 206 volunteers—a mix of psychologists and economists, students and professors, none of whom were involved in the SSRP itself
  - tqi 5y ago
    Also to be specific, none of the volunteers needed to consult an Oracle. The factors they cite (low N, "newsworthiness") are also the same factors that would give a layperson pause.
    - h0l0cube 5y ago
      I think if we're invoking the reproducabilty crisis, it's a bit ironic to then dismiss demographics
      - tqi 5y ago
        No, not really. The point is that it's possible to relibly predict whether a study will reproduce based solely on the characteristics of the study.
        h0l0cube 5y ago
        And a high portion of the cohort of that study were people that read academic papers as an occupation. I'm not saying that the study couldn't generalize to lay people, just that you haven't presented evidence of it (I believe one of the early controversies that sparked the reproducability crisis was the discovery that an excess of studies were using college students as their participants, skewing their results)
ALittleLight 5y ago
This quiz will let you test how accurate you are at predicting which psych papers replicated or not. Last time I took it, I think I was at 8 or 9 out of 10. I think something close to 50% of such papers don't replicate - so it's probably a reasonable prior to just assume any given social psychology study won't replicate. https://80000hours.org/psychology-replication-quiz/ https://80000hours.org/psychology-replication-quiz/
- h0l0cube 5y ago
  > so it's probably a reasonable prior to just assume any given social psychology study won't replicate I'd pay that. Though in this case it's not people reading the paper, but reading the article about the paper. (On reading the paper, the findings are less conclusive than the click-baity headline, but that's almost always the case)
deleted 5y ago
[deleted]

reedjosh 5y ago

It's not about an Oracle. It's that the mentioning of a topic prior to some conclusion generating observation has practically never generated replicable results. Still, these studies make news all the time because they're easy to p-hack into some interesting conclusion.

h0l0cube 5y ago
Reading the study it appears they went to some pains in not revealing their intentions: > The caregiver and child were seated across from one another at a table. The child completed an unrelated experiment with the researcher, while the caregiver completed the Scarcity or Control manipulation survey on a tablet. When the caregiver had finished, the researcher left the room under the guise of loading a second survey onto the iPad, leaving the caregiver and child alone with a toy the experimenter happened to offer as she left. A video camera and/or tape recorder recorded their interactions. The conclusion could be that being reminded or queried about hardships has a chilling effect

tqi 5y ago

People are good at predicting this: https://www.theatlantic.com/science/archive/2018/08/scientists-can-collectively-sense-which-psychology-studies-are-weak/568630/ https://www.theatlantic.com/science/archive/2018/08/scientis...

h0l0cube 5y ago
> People To be specific: > They recruited 206 volunteers—a mix of psychologists and economists, students and professors, none of whom were involved in the SSRP itself
- tqi 5y ago
  Also to be specific, none of the volunteers needed to consult an Oracle. The factors they cite (low N, "newsworthiness") are also the same factors that would give a layperson pause.
  - h0l0cube 5y ago
    I think if we're invoking the reproducabilty crisis, it's a bit ironic to then dismiss demographics
    - tqi 5y ago
      No, not really. The point is that it's possible to relibly predict whether a study will reproduce based solely on the characteristics of the study.
      - h0l0cube 5y ago
        And a high portion of the cohort of that study were people that read academic papers as an occupation. I'm not saying that the study couldn't generalize to lay people, just that you haven't presented evidence of it (I believe one of the early controversies that sparked the reproducability crisis was the discovery that an excess of studies were using college students as their participants, skewing their results)

ALittleLight 5y ago

This quiz will let you test how accurate you are at predicting which psych papers replicated or not. Last time I took it, I think I was at 8 or 9 out of 10. I think something close to 50% of such papers don't replicate - so it's probably a reasonable prior to just assume any given social psychology study won't replicate. https://80000hours.org/psychology-replication-quiz/ https://80000hours.org/psychology-replication-quiz/

h0l0cube 5y ago
> so it's probably a reasonable prior to just assume any given social psychology study won't replicate I'd pay that. Though in this case it's not people reading the paper, but reading the article about the paper. (On reading the paper, the findings are less conclusive than the click-baity headline, but that's almost always the case)

deleted 5y ago

[deleted]