5 ms·

Can someone help me understand the concern about p-value hacking? One of the comments below references this XKCD comic [0], which IIUC is an example of p-hacki

Can someone help me understand the concern about p-value hacking? One of the comments below references this XKCD comic [0], which IIUC is an example of p-hacking. But in that comic, the only difference I notice between the original hypothesis (jelly beans cause acne) and the p-hacked hypothesis (green jelly beans cause acne) is whether or not the hypothesis occurred to the researcher at the beginning of the study. And I don't understand why that would bear on the importance of each hypothesis. [0] https://xkcd.com/882/ https://xkcd.com/882/

DangitBobby 7y ago
We already expect data to sometimes indicate significant evidence against the null hypothesis when the null hypothesis is actually correct. That's just the nature of the way we structure our statistics practices. But for each additional set of variables you have and could test for relationships, you increase your chances of finding a strong relationship that appears to be significant due to random chance, but was in fact not. (You may notice that this is just a different flavor of the issues that arise from the way researchers use significance tests). The reason you need to pick your hypothesis ahead of time is because otherwise you may use intuition or just your eyeballs to find relationships in the data that are only there due to random chance. This will supposedly increase the probability of detecting spurious relationships and erroneously rejecting the null.
- DoofusOfDeath 7y ago
  Thanks, although I must admit I'm having some trouble following your logic. IIUC, you're pointing out that when examining the empirical data gathered during an experiment, it's often possible to find some identifiable subset of the data that are consistent with a refined version of the original hypothesis. E.g., maybe jellybeans in general don't correlate with acne, but green ones do. Assuming that the experiment is part of a larger effort to build or refine some model, I don't see the problem. Suppose that someone refrained from p-hacking during the first run of that experiment. IIUC, they'd look at the experimental results, and wonder if they had missed some "X" factor. So they might conject that jellybean color was relevant, and rerun the experiment with the hypothesis, "consumption of jellybeans, but only of a particular color, correlates with acne." And (assuming sample sizes were big enough), the data gathered during that second experiment would likely confirm that green jellybeans correlate with acne. But what's the point of having run that second experiment, when they could have just reached the same conclusion by testing additional hypotheses from the first experiment's data? It seems like regardless of whether you just ran one experiment and did "p-hacking", or instead ran a follow-on experiment, you end up with the same refinements to the model you're working on.

DangitBobby 7y ago

We already expect data to sometimes indicate significant evidence against the null hypothesis when the null hypothesis is actually correct. That's just the nature of the way we structure our statistics practices. But for each additional set of variables you have and could test for relationships, you increase your chances of finding a strong relationship that appears to be significant due to random chance, but was in fact not. (You may notice that this is just a different flavor of the issues that arise from the way researchers use significance tests). The reason you need to pick your hypothesis ahead of time is because otherwise you may use intuition or just your eyeballs to find relationships in the data that are only there due to random chance. This will supposedly increase the probability of detecting spurious relationships and erroneously rejecting the null.

DoofusOfDeath 7y ago
Thanks, although I must admit I'm having some trouble following your logic. IIUC, you're pointing out that when examining the empirical data gathered during an experiment, it's often possible to find some identifiable subset of the data that are consistent with a refined version of the original hypothesis. E.g., maybe jellybeans in general don't correlate with acne, but green ones do. Assuming that the experiment is part of a larger effort to build or refine some model, I don't see the problem. Suppose that someone refrained from p-hacking during the first run of that experiment. IIUC, they'd look at the experimental results, and wonder if they had missed some "X" factor. So they might conject that jellybean color was relevant, and rerun the experiment with the hypothesis, "consumption of jellybeans, but only of a particular color, correlates with acne." And (assuming sample sizes were big enough), the data gathered during that second experiment would likely confirm that green jellybeans correlate with acne. But what's the point of having run that second experiment, when they could have just reached the same conclusion by testing additional hypotheses from the first experiment's data? It seems like regardless of whether you just ran one experiment and did "p-hacking", or instead ran a follow-on experiment, you end up with the same refinements to the model you're working on.