5 ms·
It seems awkward that p-value does not account for the setting in which the test is performed. That is, if I test 1000 people at a high school football game in
by nkurz 13y ago
It seems awkward that p-value does not account for the setting in which the test is performed.
That is, if I test 1000 people at a high school football game in Peoria, Illinois, and one of them comes out positive (p=0.01), there is a very different likelihood that I have found the true president than if I test 1000 people at a official dinner in the White House and find a match (also p=0.01). In fact, I think I'd be willing to wager more on the chance that the individual tested in the White House is actually the president than the random football fan in Peoria, even if only a single person in Peoria was tested (p=0.00001).
Of course, this is only an issue if p-value is being used to express relative confidence in a conclusion, which it shouldn't (?) be. Still, how does a frequentist account for a choice of venue like this?
- vbs_redlof 13y agoP-values are not comparable across different null hypotheses, they are a property of a particular sample given a null hypothesis. You can compare p-values only by making very strong assumptions that the samples come from the same data-generating process. In almost all cases this is unlikely. A frequentist usually builds a more descriptive model by adding more variables (accounting for obvious factors that contribute to differences in observed frequencies) and increasing sample size to increase statistical power (reduce false negatives). This is not applicable in this case because there is only one president and the tests have low power. The test is bogus because you can't 'sample' and determine to probability of X==president effectively.
- klodolph 13y agoFrequentist approach would be to design an experiment, determine rules for interpreting data, and then consider the probability that the experiment lead to the correct conclusion. In this case, the experiment is really "who is president?" and not "is person X president?" The "who is president?" question does not really have a null hypothesis, so it makes no sense to talk about p-values. Instead, we can talk about two different results: identified the president correctly, and identified the president incorrectly. We then have to design the experiment in such a way that probabilities can be calculated. But this is not possible if you say the repeated experiment is testing a bunch of people to see if one is president--because the probability depends on who actually is president, and that's not known. So you consider the experiment to be "the president goes about their business and ends up in some random place, we get amnesia, and then run the test." We can simulate this experiment because we can come up with some probability distribution for where the president is. We can't come up with a probability for who the president is, because that's unknown and either 0% or 100%--only Bayesians would let you do that. Then you can choose the order of people to test to maximize the probability that the president will be identified correctly. And then you say things like, * The test had a 92% chance of succeeding. * We only had to test 14 people, and in such cases, the test had a 98% chance of succeeding.