5 ms·

As the paper points out that's accounted for in the analysis, because shockingly enough to most internet commentators researchers know that they need to control

As the paper points out that's accounted for in the analysis, because shockingly enough to most internet commentators researchers know that they need to control for confounding variables https://www.bmj.com/content/387/bmj-2024-082194 https://www.bmj.com/content/387/bmj-2024-082194

jrflo 2mo ago
Always insane to me how people will just read the headline of an article and point out "flaws" based on what they assume the article says
- cromka 2mo ago
  People jump the gun to show their perceived intellectual superiority. Typically same kind of people are last to admit of a mistake.
  - somenameforme 2mo ago
    It's not necessarily about intellectualism. Science in modern times is plagued by publish or perish. Researchers just need to get things published to keep working, and negative results don't get published. So I think people often publish stuff without proper controls/missing confounding variables/inversion of causality/etc not out of ignorance, but because they know the conclusion is dubious - but if they don't publish, they're going to be out of a job. I don't think many realize how huge the replication crisis is. You have 75% of studies in a leading social psychology journal failing to replicate. [1] Another replication effort on preclinical medical studies found only 11% were able to be replicated, a replication effort on cancer studies found the average effect size to be 85% lower than published, and much more. If you assume most science headlines, at least outside of the traditional hard sciences, are false - you're substantially more likely to be right than wrong. [1] - https://en.wikipedia.org/wiki/Replication_crisis#In_psychology https://en.wikipedia.org/wiki/Replication_crisis#In_psycholo...
    - disgruntledphd2 2mo ago
      > It's not necessarily about intellectualism. Science in modern times is plagued by publish or perish. Researchers just need to get things published to keep working, and negative results don't get published. So I think people often publish stuff without proper controls/missing confounding variables/inversion of causality/etc not out of ignorance, but because they know the conclusion is dubious - but if they don't publish, they're going to be out of a job. This is very true. In my PhD research, I managed to perfectly predict the placebo effect using a random forest. Now, this was actually due to sampling variance, and the finding went away when I averaged over splits. I often wonder if I'd still been in academia when this happened, if I would have been able to resist the siren call of a Nature/Science paper (and the consequent career benefits).
    - cromka 2mo ago
      Oh wow, I did not know that. Thanks for the input. I doubt OP's jaded response was alluding to your insight, but still thanks for that, it's genuinely good to know.
- Nasrudith 2mo ago
  They got trained to said assumptions by bad sensationalist 'science journalism'. Remember how power lines caused brain damage in children? Not through the magnetic fields or anything like that, but the lowered property values.
- Melatonic 2mo ago
  You actually read the link ?
- usef- 2mo ago
  And then their comment also reliably becomes the top-voted comment in the post
jodrellblank 2mo ago
> "researchers know that they need to control for confounding variables" 'Why ‟controlling for a variable” doesn't (usually) work' - https://dynomight.net/control/ https://dynomight.net/control/ Other things that don't work, including four other 'controlling for a variable' entries at items 39 through 43: https://dynomight.net/things/ https://dynomight.net/things/
- mcmoor 2mo ago
  Also https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0152719 https://journals.plos.org/plosone/article?id=10.1371/journal... and https://slatestarcodex.com/2014/04/26/stop-confounding-yourself-stop-confounding-yourself/ https://slatestarcodex.com/2014/04/26/stop-confounding-yours...
  - vintermann 2mo ago
    About that old Slate Star Codex article, I find it amazing that no one in that 70+ comment section of presumably smart and Rational people, bothered to look up to see if there were interventional studies on bullying. Dan Olweus did several of them in the 70s and 80s, and fought to his dying day against programs which didn't have casual intervention studies backing them.
mewpmewp2 2mo ago
The 1/60 to 1/100 difference could be explained by just one small confounder that is impossible to cover for. Therefore in my opinion making any conclusions based on this study wouldn't be fruitful at all.

jrflo 2mo ago

Always insane to me how people will just read the headline of an article and point out "flaws" based on what they assume the article says

cromka 2mo ago
People jump the gun to show their perceived intellectual superiority. Typically same kind of people are last to admit of a mistake.
- somenameforme 2mo ago
  It's not necessarily about intellectualism. Science in modern times is plagued by publish or perish. Researchers just need to get things published to keep working, and negative results don't get published. So I think people often publish stuff without proper controls/missing confounding variables/inversion of causality/etc not out of ignorance, but because they know the conclusion is dubious - but if they don't publish, they're going to be out of a job. I don't think many realize how huge the replication crisis is. You have 75% of studies in a leading social psychology journal failing to replicate. [1] Another replication effort on preclinical medical studies found only 11% were able to be replicated, a replication effort on cancer studies found the average effect size to be 85% lower than published, and much more. If you assume most science headlines, at least outside of the traditional hard sciences, are false - you're substantially more likely to be right than wrong. [1] - https://en.wikipedia.org/wiki/Replication_crisis#In_psychology https://en.wikipedia.org/wiki/Replication_crisis#In_psycholo...
  - disgruntledphd2 2mo ago
    > It's not necessarily about intellectualism. Science in modern times is plagued by publish or perish. Researchers just need to get things published to keep working, and negative results don't get published. So I think people often publish stuff without proper controls/missing confounding variables/inversion of causality/etc not out of ignorance, but because they know the conclusion is dubious - but if they don't publish, they're going to be out of a job. This is very true. In my PhD research, I managed to perfectly predict the placebo effect using a random forest. Now, this was actually due to sampling variance, and the finding went away when I averaged over splits. I often wonder if I'd still been in academia when this happened, if I would have been able to resist the siren call of a Nature/Science paper (and the consequent career benefits).
  - cromka 2mo ago
    Oh wow, I did not know that. Thanks for the input. I doubt OP's jaded response was alluding to your insight, but still thanks for that, it's genuinely good to know.
Nasrudith 2mo ago
They got trained to said assumptions by bad sensationalist 'science journalism'. Remember how power lines caused brain damage in children? Not through the magnetic fields or anything like that, but the lowered property values.
Melatonic 2mo ago
You actually read the link ?
usef- 2mo ago
And then their comment also reliably becomes the top-voted comment in the post

jodrellblank 2mo ago

> "researchers know that they need to control for confounding variables" 'Why ‟controlling for a variable” doesn't (usually) work' - https://dynomight.net/control/ https://dynomight.net/control/ Other things that don't work, including four other 'controlling for a variable' entries at items 39 through 43: https://dynomight.net/things/ https://dynomight.net/things/

mcmoor 2mo ago
Also https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0152719 https://journals.plos.org/plosone/article?id=10.1371/journal... and https://slatestarcodex.com/2014/04/26/stop-confounding-yourself-stop-confounding-yourself/ https://slatestarcodex.com/2014/04/26/stop-confounding-yours...
- vintermann 2mo ago
  About that old Slate Star Codex article, I find it amazing that no one in that 70+ comment section of presumably smart and Rational people, bothered to look up to see if there were interventional studies on bullying. Dan Olweus did several of them in the 70s and 80s, and fought to his dying day against programs which didn't have casual intervention studies backing them.

mewpmewp2 2mo ago

The 1/60 to 1/100 difference could be explained by just one small confounder that is impossible to cover for. Therefore in my opinion making any conclusions based on this study wouldn't be fruitful at all.