3 ms·
I found this paper to be quite interesting, but I have two issues with it. 1. At least at the beginning, it focuses excessively on the historical aspects of st
by amluto 12y ago
I found this paper to be quite interesting, but I have two issues with it.
1. At least at the beginning, it focuses excessively on the historical aspects of statistics. For example, it says that "most applied researchers are unmindful of the historical development of methods of statistical
inference, and of the conflation of Fisherian a
nd Neyman–Pearson ideas." To me, statisticians shouldn't /have/ to understand the history at all. For example, as a physicist, there is absolutely no need for me to understand the evolution of Ampère' theories, Faraday's theories, Maxwell theories, etc. to apply the laws of electricity and magnetism correctly.
2. The difference between p and alpha is central to the paper, but it doesn't seem to have a cogent explanation of what that difference is. (It's very clear who advocated for one and who advocated for the other, but that's not why the difference is important.)
- baghira 12y ago1. I would argue that this is a paper that intentionally goes into detail with regard to the history of statics. After all the main problem here is not how to compute p-values, nor the choice of the threshold (well, that may very well be a problem, but it's a different one) but rather the interpretation, and history is often important for interpretation, even in physics. If one wants a really compact way to convince people that the way they are thinking about p-values is wrong I'd go with the point raised by Steven Goodman in "A Dirty Dozen: Twelve P-Value Misconceptions": since the p-value is computed under the assumption that the null hypothesis is true it cannot be the probability that the null hypothesis is true, by definition. 2. As far as what the difference is, I have to admit I have not found a memorable phrase to explain it. Both Goodman in "Toward Evidence-Based Medical Statistics" and this paper give a similar wording of the difference, which i would rephrase as follows. The p-value works by inference, taking the data and assigning a probability to that data, not to an hypothesis. It is to be used to corroborate our disbelief in the null, i.e. informally. To use inference to test hypothesis one must use Bayes' Theorem, i.e. Bayes factors, and introduce prior probabilities. Hypothesis testing à la Neyman and Pearson is deductive process, in which one does assign probabilities to the hypotheses, paying the price of only being able to minimize the errors commited, not to draw inferences. One should also confront "In Fisher’s approach the researcher sets up a null hypothesis that a sample comes from a hypothetical infinite population with a known sampling distribution." with "Neyman–Pearson results are predicated on the assumption of repeated random sampling from a defined population." Given the natural predisposition by students to works in an inferential manner, it may be wise to bite the bullet and teach Bayes factor instead of the p-value cargo cult (this being a pedagogical choice, not an assessment of frequentism vs. bayesianism or a critique of p-values as conceived by Fischer).
- capnrefsmmat 12y agoThe difference between p values and alpha levels is a bit subtle, and when I first read this paper (while preparing my book, Statistics Done Wrong) it took me a while to figure out. Here's the idea. If you set alpha = 0.05, you will declare statistically significant any result that gets a p value of 0.05 or less. When there is no statistically significant difference to be found, you will have a 5% chance of falsely detecting one. But crucially, this applies on average to all tests you conduct with this alpha level. Even if an individual test gets p = 0.000001 or p = 0.04, the overall false positive rate will be 5%. More succinctly, it doesn't make sense to ask for the false positive rate of a single test. What does that even mean? You can only ask for the false positive rate of a procedure you use many times. So you can't get p = 0.01 and declare this means you have a false positive rate of 1%.
- philh 12y agoPossibly worth clarifying: the false positive rate (https://en.wikipedia.org/wiki/False_positive_rate https://en.wikipedia.org/wiki/False_positive_rate) is "probability that a test will return positive, conditional on the hypothesis being false". It's the rate of false positives within the set of negatives, not the rate of false positives within all tests.