5 ms·

Psych study with n=114. Given the reproducibility crisis in psychology, I'll wait for the replication.

madrox 4y ago
I tend to agree. While I think the sample size is fine, the way perceived positivity and perceived pleasure were measured seems pretty subjective in a way I find hard to generalize. The paper does seem awfully suggestive, though. Worthy of more exploration for sure.
FastMonkey 4y ago
All undergrads from a single university also. Just based on the social development of people as they go through undergrad, I'd guess that the variance in behavioural change before and after treatment is fairly high whether they're treated or not.
- snovv_crash 4y ago
  Just like a lot of pharma papers need a `... in mice` appended to their titles, a lot of psych papers need `... in undergrads` appended to theirs.
raddan 4y ago
Sure, do that. But also: > The power analysis used a power criterion of (1 − β) = 0.80, which indicated that a mean cell size of n = 54 was sufficient to replicate significant effects of acetaminophen. The question is not whether n=114 is "large enough" because that question, by itself, does not make sense. The question is whether n=114 is "large enough for the observed effect." Given that the p-values for the "personal pleasure" and "empathetic feelings" measures are under 0.001, the answer is "yes," potential methodological issues notwithstanding.
- deleted 4y ago
  [deleted]
- teaearlgrey_hot 4y ago
  Disagree from experience. I find it far more likely that they considered how to skew the p-values low enough to where they become significant. “Personal pleasure” & “Empathetic feelings” do NOT sound like terms which correlate to reproducibility to me.
  - HWR_14 4y ago
    > “Personal pleasure” & “Empathetic feelings” do NOT sound like terms which correlate to reproducibility to me. All studies that use terms like that have a section that explains how they are measuring it, how other studies have used the same measure and how that measure itself is tested for reproducibility and accuracy.
    - ngvrnd 4y ago
      doesn't necessarily mean they are ACTUALLY measuring anything real. just sayin'
      - HWR_14 4y ago
        So without training in the field, having read about the specific example or anything else, you can say the entire study is trash?
        teaearlgrey_hot 4y ago
        harry8 4y ago
        Pretty sure caution was what was advised with no opinion one way or the other. Being extra cautious about preliminary psych findings is only sensible. It might be considered batshit crazy to do anything else given the issues we've found out about in that field. Does it reproduce? Is the right question to ask.
        captainbland 4y ago
        I'd think of it more like such studies being potential stepping stones towards the establishment of scientific facts, rather than establishing those facts alone. But it will take more work and by separate groups before anyone can have faith that these findings are meaningful - especially given the background of the reproducibility crisis.
mattkrause 4y ago
I don't think you should sneer at a triple-digit sample size that's justified with a power analysis based on their previous work. I agree that you'd want to see replications before making any major decisions--these effects are often finnicky--but this seems to me like a reasonable-sized step forward.
- simulate-me 4y ago
  They also had a prior justification for running this experiment based on known effects of Tylenol on the brain.
plsbenice34 4y ago
Seems like every time now, when I read a medical study, especially psychiatry-related, I'm shocked that it could even be published. I have no training in the field, so nobody would take me seriously and I even doubt myself, but the study designs just seem so poor
amelius 4y ago
Scientist: Hey, that's odd ... HN reader: I'll wait for the replication.
arcticbull 4y ago
There's been a few over the years. [1] 2016: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5015806/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5015806/ [2] 2015: https://journals.sagepub.com/doi/abs/10.1177/0956797615570366 https://journals.sagepub.com/doi/abs/10.1177/095679761557036... [3] 2012: http://www.psychologicalscience.org/news/releases/broken-hearts-really-hurt.html http://www.psychologicalscience.org/news/releases/broken-hea...

I tend to agree. While I think the sample size is fine, the way perceived positivity and perceived pleasure were measured seems pretty subjective in a way I find hard to generalize. The paper does seem awfully suggestive, though. Worthy of more exploration for sure.

FastMonkey 4y ago

All undergrads from a single university also. Just based on the social development of people as they go through undergrad, I'd guess that the variance in behavioural change before and after treatment is fairly high whether they're treated or not.

snovv_crash 4y ago
Just like a lot of pharma papers need a `... in mice` appended to their titles, a lot of psych papers need `... in undergrads` appended to theirs.

raddan 4y ago

Sure, do that. But also: > The power analysis used a power criterion of (1 − β) = 0.80, which indicated that a mean cell size of n = 54 was sufficient to replicate significant effects of acetaminophen. The question is not whether n=114 is "large enough" because that question, by itself, does not make sense. The question is whether n=114 is "large enough for the observed effect." Given that the p-values for the "personal pleasure" and "empathetic feelings" measures are under 0.001, the answer is "yes," potential methodological issues notwithstanding.

deleted 4y ago
[deleted]
teaearlgrey_hot 4y ago
Disagree from experience. I find it far more likely that they considered how to skew the p-values low enough to where they become significant. “Personal pleasure” & “Empathetic feelings” do NOT sound like terms which correlate to reproducibility to me.
- HWR_14 4y ago
  > “Personal pleasure” & “Empathetic feelings” do NOT sound like terms which correlate to reproducibility to me. All studies that use terms like that have a section that explains how they are measuring it, how other studies have used the same measure and how that measure itself is tested for reproducibility and accuracy.
  - ngvrnd 4y ago
    doesn't necessarily mean they are ACTUALLY measuring anything real. just sayin'
    - HWR_14 4y ago
      So without training in the field, having read about the specific example or anything else, you can say the entire study is trash?
      - teaearlgrey_hot 4y ago
      - harry8 4y ago
        Pretty sure caution was what was advised with no opinion one way or the other. Being extra cautious about preliminary psych findings is only sensible. It might be considered batshit crazy to do anything else given the issues we've found out about in that field. Does it reproduce? Is the right question to ask.
      - captainbland 4y ago
        I'd think of it more like such studies being potential stepping stones towards the establishment of scientific facts, rather than establishing those facts alone. But it will take more work and by separate groups before anyone can have faith that these findings are meaningful - especially given the background of the reproducibility crisis.

mattkrause 4y ago

I don't think you should sneer at a triple-digit sample size that's justified with a power analysis based on their previous work. I agree that you'd want to see replications before making any major decisions--these effects are often finnicky--but this seems to me like a reasonable-sized step forward.

simulate-me 4y ago
They also had a prior justification for running this experiment based on known effects of Tylenol on the brain.

plsbenice34 4y ago

Seems like every time now, when I read a medical study, especially psychiatry-related, I'm shocked that it could even be published. I have no training in the field, so nobody would take me seriously and I even doubt myself, but the study designs just seem so poor

amelius 4y ago

Scientist: Hey, that's odd ... HN reader: I'll wait for the replication.

arcticbull 4y ago

There's been a few over the years. [1] 2016: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5015806/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5015806/ [2] 2015: https://journals.sagepub.com/doi/abs/10.1177/0956797615570366 https://journals.sagepub.com/doi/abs/10.1177/095679761557036... [3] 2012: http://www.psychologicalscience.org/news/releases/broken-hearts-really-hurt.html http://www.psychologicalscience.org/news/releases/broken-hea...