4 ms·
I'm going to go for the embarrassing question, but I don't understand p-values or how to design a study. Any pointers gratefully recvd If I understand correct
by lifeisstillgood 12y ago
I'm going to go for the embarrassing question, but I don't understand p-values or how to design a study. Any pointers gratefully recvd
If I understand correctly, let's say I am concerned Earths gravity is increasing underneath Kent school buildings this stunting kids growth.
I decide to test this by making a statistical survey with a p-value in it.
We shall sample the heights of 10,000 randomly chosen children from 10 counties, one of which is Kent (so 1,000 kids from each school district)
My null hypothesis is that there is no gravitational anaomloly and so the height distribution should be equal or the mean height should be equal or ...
So I get a bit lost here.
What is 95% significance ...
- thaumasiotes 12y agoGoing by the wikipedia article, a p-value is the probability of getting a result "at least as extreme" as what you observe under some hypothesis (in practice, the null hypothesis, however defined). Imagine we're spinning a spinner to get one of two values ("red" or "blue"), and we think a good null hypothesis is "red and blue are equally likely outcomes of this spinner". Spinning 500 times, we "should" see 250 of each, but obviously the odds of that are quite low. So if I spin the spinner 500 times and see 400 reds and 100 blues, what does that mean? Under the null hypothesis, the odds of that result are (0.5)^400 * (0.5)^100 * (500 choose 400). The first step in calculating the p-value is to decide which outcomes count as "at least as extreme"; the obvious candidates are (1) any result with 400 or more reds; and (2) any result with 400 or more of either color. (1) is better for talking about the hypothesis "this spinner is biased towards red", and (2) is better for talking about the hypothesis "this spinner is biased". Obviously, under the null hypothesis, the p-value calculated for (2) will be twice as large as the p-value calculated for (1), which will have implications for the publishability of your results, so you should choose (1) and, if necessary, adjust your hypothesis to suit. ;) Anyway, in principle, you add up the odds of all the outcomes that count as "at least as extreme" as the result you observed, and that total is your p-value. You can report "under the null hypothesis, there is less than a 5% chance of observing this result" (p < 0.05), and the person after you in the chain can report "it is at least 95% likely that the spinner is secretly Republican", and the press can handle things from there. Now, you've got a very interesting example going on in your question, because your null hypothesis specifies that the height distribution of children in a county is due to the local gravity. (Is this a reasonable null hypothesis? No.) So if you do your survey and find that Kent kids are taller, and the difference in mean height is significant at the p < 0.03 level, you can say "we reject the null hypothesis with 97% confidence". The predictive part of your null hypothesis says "the height distributions are identical in every county", and that's pretty easy to calculate about. Your rejection is perfectly justifiable! But, a reporter could then look at the phrasing of the non-predictive part of your hypothesis and report "97% chance of a gravitational anomaly in Kent", which would be unjustifiable; if Kent kids are taller, I'd look at factors like nutrition and genetics long before I even considered the possibility of a gravitational anomaly. The lesson here is that rejecting the null hypothesis "there is no gravitational anomaly and therefore the height distributions of children in every county will be identical" doesn't allow you to conclude "what do you know, there is a gravitational anomaly after all!", even though the rejection was fairly legitimate.
- lifeisstillgood 12y agoSo the nice diagram with a small area of 2nd std Dev filled in is showing all the results of throwing a dice that get a result at least as extreme as the observed result. So if I throw a dice 60 times I would expect a fair dice to come up with 10 1's, 10 2's etc. if it came up with 50 1's I could say that I reject the fair dice hypothesis with a confidence of (Thinking)
- stdbrouw 12y agoImagine that your sample of kids in Kent are actually 5mm smaller on average than your samples from the 9 other counties. The question to then ask is: how likely is it that this would happen purely due to chance? After all, even a fair die will occasionally throw tens or even hundreds of sixes in row. Likewise, your Kent sample might just be a bit odd. A p-value answers this question: imagine that there's no gravitational effect, how often would your sample of children in Kent end up 5mm smaller anyway due to chance? In most cases, the lower this number, the better, because it means something other than chance might be at play. If your p-value were 0.95, you'd conclude that even though the sample of Kent kids is smaller, this would have happened purely due to chance 95 out of a 100 times, so there's not likely to be a real difference. Of course, with samples as large as those you mention, your p-value is pretty much guaranteed to be low, and hence you're guaranteed to have a statistically significant effect. It's probably easier to understand in the context of much smaller sample sizes: if you play three tennis matches against a friend, and he wins 2 out of 3, what's the chances he's actually better than you, and what are the chances that he just got lucky this time? Turns out, it's 50/50. That's what a p-value is trying to get at. Which is one of the big reasons why p-values get so much (deserved) hate: they're influenced heavily by both effect size and sample size, so you can't tell one from the other.