2 ms·
The problem with all of this is that the reducto-scientific paradigm for understanding science has long been extended to communicating and teaching about scienc
by avs733 9y ago
The problem with all of this is that the reducto-scientific paradigm for understanding science has long been extended to communicating and teaching about science.
As the author does a really nice job explaining, and as seen in the Fisher quote, these articulations of heuristics and guidelines are often taken as closed form rules.
Look beyond statistics and you see it everywhere. Common ones include entrepreneurship and design...in both areas, experts' ways of thinking are often highly situated, highly metacognitive, and the actions they take are inherently inseparable from their thinking process. However, because of the academic drive towards objective/deterministic/observable phenomenon the research tends to report and attribute only the actions. The result is that those actions, rather than the underlying thinking processes, are valued and taught.
The result is simulations of expertise masquarading as knowledge. Its one thing when its students, but as you are seeing in psychology's 'replication crisis' (which, side note, is kind of a metaversion of its own critique) it can create real problems when surface level understanding is accepted and generalized as a normative 'truth' in a field. You see it in economics and business a lot...strive to appear scientific, but do so in ways that inherently betray the underlying structure of what you are studying. It comes from an underlying value in those communities, and society, that the only truth is objective truth.
If I have an experiment where I am screening 5 possible predictors and I get p values of .9 for 4 of them and .52 for 1...I would be an idiot not to pursue the 1. if I get 4 .49s and 1 .00000000001...same thing. Statistics is relative, literally.
[happy to provide citations...not sure anyone really cares]
- thedailymail 9y agoI care, and agree with your points. It seems at least part of the fetishization of p < 0.05 has arisen from its common use as the cutoff point in drug clinical trials, and thus represent a make or break point in a multi-billion dollar industry. There needs to be some predefined standard for such trials to prevent cherry-picking and other games, but as the blog describes, this could have just as easily been a different threshold. Similar observations can also be made about the values used to set study sample size (based on somewhat arbitrary alpha and beta). The arbitrariness at the heart of the regulatory enterprise may seem disconcerting, but the alternative (no shared standards for evaluating drug efficacy) has also been tried historically, and the result was markets flooded with useless, often dangerous, products many of which nonetheless sold very well.
- avs733 9y ago> It seems at least part of the fetishization of p < 0.05 has arisen from its common use as the cutoff point in drug clinical trials, and thus represent a make or break point in a multi-billion dollar industry. I think that is a bit recursive. Its not so much a fetishization as it is a misunderstanding that results in it being a valuable target. It would be a heck of a lot more useful as a target (and granted...I would argue the target should be more like .001) if more research adopted Bayesian statistical techniques where you can't as easily P hack.