9 ms·
To follow that up (so people know what they actually are), what p-values represent is the likelihood we would observe our data, given the null hypothesis. Sett
by comte7092 4y ago
To follow that up (so people know what they actually are), what p-values represent is the likelihood we would observe our data, given the null hypothesis.
Setting a cutoff of .05 is saying “if there’s less than a 5% chance we’d see this data, assuming the null hypothesis, then we can assume that the null hypothesis is false”
- nickff 4y agoBut this statement only applies to a 'naive' (or first) statistical analysis or test on the dataset. Once the researcher starts changing their assumptions in response to the results they're seeing, they're p-hacking and p-values are no longer meaningful. In addition, once you have multiple researchers looking at the same dataset with different assumptions, and you factor in publication bias, the p-value also loses meaning.
- comte7092 4y agoWell, yes and no. The p-value still means the same thing, but when you take a dataset and go looking for any result that is under a certain threshold, you’ll probably find it. “Unlikely” events happen all the time! What your comment is highlighting is an issue with bad experimental design. (And, obviously, with our publication regime)