23 ms·

Too many important missing details, so it is hard to assess the validity of the results. Generally, observational studies of this kind are hotbed for false alar

by drgo 2y ago

Too many important missing details, so it is hard to assess the validity of the results. Generally, observational studies of this kind are hotbed for false alarms and false hopes. The difficulty is that on average the people who sleep well and are physically active are also the people who eat well, have higher incomes and better access to healthcare. So it is hard to isolate one factor as the most responsible for the observed outcomes (the problem of confounding). Also, even if the observed average effects are real, it is hard to predict how that would translate into an effect at the individual level because the effect may depend on age, gender, genes, frailty, environmental factors etc (the problem of effect modification or interaction). Finally, because no actual data is shown, we do not know reliable and reproducible the results are (the problem of random error). Be careful of drawing any conclusions from observational studies especially studies of unknown methods and no published results. Edited for typos

deleted 2y ago
[deleted]
uoaei 2y ago
> The difficulty is that on average the people who sleep well and are physically active are also the people who eat well, have higher incomes and better access to healthcare Truly effective causal studies are still in their infancy, relatively speaking, and are still prone to the same correlation//causation fallacies. For instance, it's no fun studying things that may be the root cause of those behaviors but is hard to quantify, like "go-getter-ness" or something similarly illegible to scientific methods. You don't really know the structure of the problem so you can't put useful priors on it and focus in toward a reasonable inference. It's much easier to run t-tests and ANOVA by writing a couple lines of R so that's where we've settled as a community.
- drgo 2y ago
  This is a very important point. It sounds obvious that the proxy for something should be treated as if it was the thing, but research is a business and researchers are people subject to peer pressure and social desirability bias. So the proxy becomes the thing.
DemocracyFTW2 2y ago
Thanks for this comment. A text like this should really be present whenever medical study results are presented on HN.
lm28469 2y ago
The thing is that you can apply that to virtually every study. There are just way too many variables to control for. Unless you have a few thousand of identical twins locked down in sterile rooms from birth you can't do much
- nkurz 2y ago
  You're not wrong. Almost all observational studies could be subject to confounders which if true would invalidate their claimed results. But you seem to be suggesting that just because science is hard we should accept the results of all these flawed studies at face value without further questioning. The better conclusion is that we should be skeptical of almost all such studies, and always look closely at the data and methodology to see if a confounding effect might contradict the conclusion. We should only give weight to studies that stand up to such scrutiny, and even then we should realize their limitations.
- drgo 2y ago
  yes...I do not trust the claims of any single observational study. Using observational studies is necessary to help understand a problem in preparation for well-conducted large randomized clinical trials (RCTs). Sometime RCTs are not ethical or feasible (like etiological studies of cancer), and then we need many observational studies in different settings and using different designs showing a large effect that cannot be explained by known or unknown confounders and supported by many experimental studies (e.g., in cells). And even then we accept the results on the principle that is better to be safe than sorry. Many observational studies claimed that HRT protect women from heart disease until the definitive trial was done and it was found HRT actually increases the risk of heart disease!! Hundreds of other examples exist of well-researched and widely accepted hypotheses that turned out to be false in RCTs. For every observational study that claims X causes Y, I could find one that claims that X protects against Y, that X has nothing to do with Y, that Y causes X, that there is no X and no Y.. etc. It is the Wild West of science.

by drgo 2y ago

deleted 2y ago

[deleted]

uoaei 2y ago

> The difficulty is that on average the people who sleep well and are physically active are also the people who eat well, have higher incomes and better access to healthcare Truly effective causal studies are still in their infancy, relatively speaking, and are still prone to the same correlation//causation fallacies. For instance, it's no fun studying things that may be the root cause of those behaviors but is hard to quantify, like "go-getter-ness" or something similarly illegible to scientific methods. You don't really know the structure of the problem so you can't put useful priors on it and focus in toward a reasonable inference. It's much easier to run t-tests and ANOVA by writing a couple lines of R so that's where we've settled as a community.

drgo 2y ago
This is a very important point. It sounds obvious that the proxy for something should be treated as if it was the thing, but research is a business and researchers are people subject to peer pressure and social desirability bias. So the proxy becomes the thing.

DemocracyFTW2 2y ago

Thanks for this comment. A text like this should really be present whenever medical study results are presented on HN.

lm28469 2y ago

The thing is that you can apply that to virtually every study. There are just way too many variables to control for. Unless you have a few thousand of identical twins locked down in sterile rooms from birth you can't do much

nkurz 2y ago
You're not wrong. Almost all observational studies could be subject to confounders which if true would invalidate their claimed results. But you seem to be suggesting that just because science is hard we should accept the results of all these flawed studies at face value without further questioning. The better conclusion is that we should be skeptical of almost all such studies, and always look closely at the data and methodology to see if a confounding effect might contradict the conclusion. We should only give weight to studies that stand up to such scrutiny, and even then we should realize their limitations.
drgo 2y ago
yes...I do not trust the claims of any single observational study. Using observational studies is necessary to help understand a problem in preparation for well-conducted large randomized clinical trials (RCTs). Sometime RCTs are not ethical or feasible (like etiological studies of cancer), and then we need many observational studies in different settings and using different designs showing a large effect that cannot be explained by known or unknown confounders and supported by many experimental studies (e.g., in cells). And even then we accept the results on the principle that is better to be safe than sorry. Many observational studies claimed that HRT protect women from heart disease until the definitive trial was done and it was found HRT actually increases the risk of heart disease!! Hundreds of other examples exist of well-researched and widely accepted hypotheses that turned out to be false in RCTs. For every observational study that claims X causes Y, I could find one that claims that X protects against Y, that X has nothing to do with Y, that Y causes X, that there is no X and no Y.. etc. It is the Wild West of science.