3 ms·
One of the problems is that if you run more studies, you are statistically more likely to eventually yield a statistically significant result. The tests that fo
by juujian 3y ago
One of the problems is that if you run more studies, you are statistically more likely to eventually yield a statistically significant result. The tests that folks run are not designed for series of statistical tests, but for individual tests. So when you run a ton of experiments and throw away all the statistically insignificant results, you are engaging in the form of p-hacking that the discipline of psychology is currently struggling with. The overarching replication crisis.
And you may be thinking that's all harmless, but it's really not. Folks running experiments are competing for jobs with other folks who might do more worthwhile research. Because psychologists are good at running the pipeline, and they have found a nearly inexhaustible source of publications (experiments run on undergrad students), the up the bar for everyone else.
I remember just a few days ago a discussion on this site about how a hiring committee skipped over the "better" applicant with more publications and citations than the successful candidate. The reality is that there are plenty of faculty on hiring committees out there who are looking for candidates with a good publication record, because that is how they assess if someone is a good researcher. Psychologists have already succeeded in pushing out most of the less experiment-prone folks in their discipline. The pressure extends to adjacent fields.
- DonsDiscountGas 3y ago100%. Theoretically one can account testing multiple hypotheses, but shockingly nobody wants to do that.