3 ms·
p-hacking is a research "dark pattern" where a researcher fits several similar models and reports only the one that has the significant p-value for the relation
by pacbard 5y ago
p-hacking is a research "dark pattern" where a researcher fits several similar models and reports only the one that has the significant p-value for the relationship of interest.
This strategy is possible because p-values are themselves stochastic and a researcher will find one significant p-value for every 20 models that they run (at least on average).
p-hacking could also refer to pushing a p-value close to the significant cut-off (usually 0.05) by modifying the statistical model slightly until the desired result is achieved. This process usually involves the inclusion of control variables that are not really related to the outcome but that will change the standard errors/p-values.
Another way to p-hack is to drop specific observations until the desired p-value is reached. This process usually involves removing participants from a sample for a seemingly legitimate reason until the desired p-value is achieved. Usually identifying and eliminating a few high leverage observations is enough to change the significance level of a point estimate.
Multiple strategies to address p-hacking have been proposed and discussed. One of the most popular ones is pre-registration of research designs and models. The idea here is that a researcher would publish their research design and models before conducting the experiment and they will report only the results from the pre-registered models. This process eliminates the "fishing expedition" nature of p-hacking.
Other strategies involve better research designs that are not sensitive to model respecification. These are usually experimental and quasi-experimental methods that leverage an external source of variation (external to both the researcher and the studied system, like random assignment to conditions) to isolate the relationship between two variables.
- pthread_t 5y agoI saw this firsthand as an undergrad research assistant in a neuroscience lab. How did it go when I brought it up? Swept under the rug and published in a high-impact journal.
- harry8 5y agoI'm sorry. I'm interested in this question: Did seeing and rasing that affect your career path thereafter? Whether by your own choice or otherwise.
- pthread_t 5y agoThe experience helped me realize that a non-trivial amount of work done in these labs is worthless and probably even harmful. They don't seem to care as much about the science as they care about publications (on the part of the PI) and being published (the PhD students). Moreover, the lab pushed me out and decided to use my work anyway. Specifically, they requested that anybody that used the software I wrote give them authorship on their papers. I'm happily employed as a software engineer now.
- andi999 5y agoI am not posting it because it is xkcd, but it visualizes the simplest trap quite nicely: https://xkcd.com/882/ https://xkcd.com/882/ (also makes you wonder what if 20 groups study this phenomena, each restiricting itself to one color)