The difference between p values and alpha levels is a bit subtle, and when I first read this paper (while preparing my book, Statistics Done Wrong) it took me a

The difference between p values and alpha levels is a bit subtle, and when I first read this paper (while preparing my book, Statistics Done Wrong) it took me a while to figure out. Here's the idea. If you set alpha = 0.05, you will declare statistically significant any result that gets a p value of 0.05 or less. When there is no statistically significant difference to be found, you will have a 5% chance of falsely detecting one. But crucially, this applies on average to all tests you conduct with this alpha level. Even if an individual test gets p = 0.000001 or p = 0.04, the overall false positive rate will be 5%. More succinctly, it doesn't make sense to ask for the false positive rate of a single test. What does that even mean? You can only ask for the false positive rate of a procedure you use many times. So you can't get p = 0.01 and declare this means you have a false positive rate of 1%.