3 ms·

You shouldn't trust any single study. Researchers are not perfect, reviewers are not perfect, the systems they observe are complex. Even in the "hard" sciences

You shouldn't trust any single study. Researchers are not perfect, reviewers are not perfect, the systems they observe are complex. Even in the "hard" sciences even if the experimental data itself is solid there can be confounders that were overlooked and will only be uncovered by follow-up studies, depending on importance of the result it might take years. That's why have meta-analysis and systematic reviews and even those aren't exactly bullet-proof.

deleted 8y ago
[deleted]
User23 8y ago
Meta analysis is just the weighted average of expert opinion. Which makes it worthless when the experts don’t actually know anything. https://jamanetwork.com/journals/jama/fullarticle/2698337 https://jamanetwork.com/journals/jama/fullarticle/2698337 You should be skeptical of any so-called science that cannot actually rigorously apply the scientific method, be it for practical, ethical, or any other reasons.
btrettel 8y ago
> Even in the "hard" sciences even if the experimental data itself is solid there can be confounders that were overlooked and will only be uncovered by follow-up studies, depending on importance of the result it might take years. I work in fluid dynamics. In one subfield I work in there is at least one confounder in most studies (Weber number and Reynolds number are confounded, in particular) and an important variable is often omitted (turbulence intensity). Many people don't seem aware that turbulence intensity matters, either, and most who do think it matters don't want to measure or estimate it, so it tends to be ignored. Just because it's hard to measure does not mean it's not important! This is supposed to be "hard" science, but sometimes I feel it's not much better than social science. How long has this been a problem? I'd say over 50 years, easily! At a recent conference, one of my papers addressed these issues briefly (I tried to avoid them in a data analysis), but I think it'll take more than that to correct these problems. So I'm planning a new article specifically addressing these concerns. I still expect progress to be slow, but any movement in the right direction would be appreciated...
- tenaciousDaniel 8y ago
  That's really interesting. Seems like good science is always going to be an uphill battle against human nature/complacency.
cultus 8y ago
Actual meta-analyses and systematic replication are so rare as to basically not matter. In addition, many actual scientists are completely unaware of these fundamental issues in science. Not to mention scientist's statistical misconceptions. The vast, vast majority (over 90% from surveys I've seen) commit the inverse-probability fallacy, for example.

deleted 8y ago

[deleted]

User23 8y ago

Meta analysis is just the weighted average of expert opinion. Which makes it worthless when the experts don’t actually know anything. https://jamanetwork.com/journals/jama/fullarticle/2698337 https://jamanetwork.com/journals/jama/fullarticle/2698337 You should be skeptical of any so-called science that cannot actually rigorously apply the scientific method, be it for practical, ethical, or any other reasons.

btrettel 8y ago

> Even in the "hard" sciences even if the experimental data itself is solid there can be confounders that were overlooked and will only be uncovered by follow-up studies, depending on importance of the result it might take years. I work in fluid dynamics. In one subfield I work in there is at least one confounder in most studies (Weber number and Reynolds number are confounded, in particular) and an important variable is often omitted (turbulence intensity). Many people don't seem aware that turbulence intensity matters, either, and most who do think it matters don't want to measure or estimate it, so it tends to be ignored. Just because it's hard to measure does not mean it's not important! This is supposed to be "hard" science, but sometimes I feel it's not much better than social science. How long has this been a problem? I'd say over 50 years, easily! At a recent conference, one of my papers addressed these issues briefly (I tried to avoid them in a data analysis), but I think it'll take more than that to correct these problems. So I'm planning a new article specifically addressing these concerns. I still expect progress to be slow, but any movement in the right direction would be appreciated...

tenaciousDaniel 8y ago
That's really interesting. Seems like good science is always going to be an uphill battle against human nature/complacency.

cultus 8y ago

Actual meta-analyses and systematic replication are so rare as to basically not matter. In addition, many actual scientists are completely unaware of these fundamental issues in science. Not to mention scientist's statistical misconceptions. The vast, vast majority (over 90% from surveys I've seen) commit the inverse-probability fallacy, for example.