4 ms·

How about: The p-value is, in a statistical hypothesis test, the probability of Type I error, that is the probability of rejecting the null hypothesis when it

by graycat 11y ago

How about: The p-value is, in a statistical hypothesis test, the probability of Type I error, that is the probability of rejecting the null hypothesis when it is true. Or, suppose we have some data and want to use it to estimate the value of some number b. It can be that if we can assume something about b, e.g., that b = 0, then we can calculate the probability distribution of our estimate of b. This assumption about b is the null hypothesis. Intuitively it is a hypothesis that there is no effect, that what we thought might have happened didn't, was a null effect. We can make two mistakes. We can reject the null hypothesis when it is true -- this is called Type I error. Or we can accept the null hypothesis when it is false -- this is called Type II error. With our null hypothesis, we get and look at the distribution of our estimate of b and see where our actual estimate is in that distribution. The p-value is the probability, from the distribution of our estimate, of getting an estimate as far or farther from our null hypothesis value for b as we did. So, if the p-value is really small, say, 1%, then we can reject the null hypothesis, that is, say that it is false, and be wrong only 1% of the time. E.g., if our null hypothesis is that b = 0 and our estimate of b is 10 and from the distribution of our estimate a value of our estimate being as far as 10 from b = 0 is 1%, then we reject that b = 0 and conclude that it b is not zero and are wrong only 1% of the time. If the probability of our estimate being greater than or equal to 10 is 1% and we reject the null hypothesis, then we conclude that b > 0. This is all just hypothesis testing in statistics 101. Will also want to know about the power of a test, the t-test, the F ratio, the chi-squared test, and resampling and distribution-free tests. There is more detail in https://en.wikipedia.org/wiki/Type_I_and_type_II_errors https://en.wikipedia.org/wiki/Type_I_and_type_II_errors

nonbel 11y ago
>"How about: The p-value is, in a statistical hypothesis test, the probability of Type I error, that is the probability of rejecting the null hypothesis when it is true." No, see here: "Of critical importance, as Goodman (1993) has pointed out, is the extensive failure to recognize the incompatibility of Fisher’s evidential p value with the Type I error rate, α, of Neyman–Pearson statistical orthodoxy." P Values are not Error Probabilities http://www.uv.es/sestio/TechRep/tr14-03.pdf http://www.uv.es/sestio/TechRep/tr14-03.pdf
- graycat 11y ago
  Okay, I looked at that PDF. I read through several pages and mostly saw lots of discussion of statements the authors claimed were common and wrong. I can agree with the wrong but am less convinced about the common. I never saw a clean definition of p-value or alpha -- probability of Type I error. In my study of statistics, p-value and alpha are essentially the same. Maybe a difference: A p-value is what a researcher has in mind before the hypothesis test as a 'cutoff value', say, 5%, for alpha while alpha is whatever the statistical hypothesis test says, for the data and the test, is the probability of Type I error, e.g., 3.141592653 or some such random number. So, since the 3.14 is less than the p-value of 5%, the experimenter rejects the null hypothsis. MAYBE this is the difference, if any, between p-value and alpha, in which case we're talking a triviality. The paper you referenced also starts to get wound up over beta = 1 - alpha as the power of a test. Okay, but that little equation is the definition of power. So,if have some data and a p-value or alpha in mind, and have several candidate statistical hypothesis tests, say, some parametric and the others distribution-free, etc., then will want to use the test with the smallest probability of Type II error, that is, the largest beta. Okay. Now are done with beta, and no more strain or struggle needed. A biggie point is the real role of the null hypothesis -- it lets us calculate some probabilities that, otherwise, we would not have assumptions enough to do. If there was a fight between Fisher and Neyman-Pearson nearly 100 years ago, then I'm sorry, but now I have no sympathy for whatever the heck, if anything, they were arguing about then.
  - nonbel 11y ago
    >"I never saw a clean definition of p-value or alpha -- probability of Type I error." On re-reading this I agree. They do not make a clean comparison. They do define them though: p-value: "the probability of the observed and more extreme data given the null" alpha: "α is the long-run frequency of Type I errors" To begin with, you appear to have these reversed. Second, the p-value is dependent on the data, it will be different from experiment to experiment. The exact same experiment can give you p=0.001 one time and p=0.32 the next time. Which is the probability of type I error?
    - 11y ago

by graycat 11y ago

nonbel 11y ago

>"How about: The p-value is, in a statistical hypothesis test, the probability of Type I error, that is the probability of rejecting the null hypothesis when it is true." No, see here: "Of critical importance, as Goodman (1993) has pointed out, is the extensive failure to recognize the incompatibility of Fisher’s evidential p value with the Type I error rate, α, of Neyman–Pearson statistical orthodoxy." P Values are not Error Probabilities http://www.uv.es/sestio/TechRep/tr14-03.pdf http://www.uv.es/sestio/TechRep/tr14-03.pdf

graycat 11y ago
Okay, I looked at that PDF. I read through several pages and mostly saw lots of discussion of statements the authors claimed were common and wrong. I can agree with the wrong but am less convinced about the common. I never saw a clean definition of p-value or alpha -- probability of Type I error. In my study of statistics, p-value and alpha are essentially the same. Maybe a difference: A p-value is what a researcher has in mind before the hypothesis test as a 'cutoff value', say, 5%, for alpha while alpha is whatever the statistical hypothesis test says, for the data and the test, is the probability of Type I error, e.g., 3.141592653 or some such random number. So, since the 3.14 is less than the p-value of 5%, the experimenter rejects the null hypothsis. MAYBE this is the difference, if any, between p-value and alpha, in which case we're talking a triviality. The paper you referenced also starts to get wound up over beta = 1 - alpha as the power of a test. Okay, but that little equation is the definition of power. So,if have some data and a p-value or alpha in mind, and have several candidate statistical hypothesis tests, say, some parametric and the others distribution-free, etc., then will want to use the test with the smallest probability of Type II error, that is, the largest beta. Okay. Now are done with beta, and no more strain or struggle needed. A biggie point is the real role of the null hypothesis -- it lets us calculate some probabilities that, otherwise, we would not have assumptions enough to do. If there was a fight between Fisher and Neyman-Pearson nearly 100 years ago, then I'm sorry, but now I have no sympathy for whatever the heck, if anything, they were arguing about then.
- nonbel 11y ago
  >"I never saw a clean definition of p-value or alpha -- probability of Type I error." On re-reading this I agree. They do not make a clean comparison. They do define them though: p-value: "the probability of the observed and more extreme data given the null" alpha: "α is the long-run frequency of Type I errors" To begin with, you appear to have these reversed. Second, the p-value is dependent on the data, it will be different from experiment to experiment. The exact same experiment can give you p=0.001 one time and p=0.32 the next time. Which is the probability of type I error?
  - 11y ago