Maybe what I wrote was misleading or vague. What I mean is that to do a study, you have some model. It's implicit in the way the data is collected, whether that

Maybe what I wrote was misleading or vague. What I mean is that to do a study, you have some model. It's implicit in the way the data is collected, whether that be experiment or observational study. But it's a model with assumptions. The problem is that all models are wrong to varying extents, but often in the scientific community the wrongness of models isn't recognized. You estimate your effect size or the p-value or posterior predictive p or whatever it is, but all those themselves are some estimates based on assumptions, and there's always some probability of being radically incorrect. My concern isn't with science per se, it's that science as a community is often too arrogant and underestimates the extent to which it is likely to be incorrect. I just think critics need to be given more respect, especially when the argument against the critics is "we did a carefully controlled lab study under isolated conditions." Maybe there needs to be a greater meta-scientific perspective with these things, like some policy that explicitly asks "how often when these types of evidence are submitted is something wrong?"