3 ms·
>There are two legitimate ways to get lower p-values: You can have a larger effect size, or you can have a larger number of samples[0]. Of course, you can't cha
by BeetleB 9y ago
>There are two legitimate ways to get lower p-values: You can have a larger effect size, or you can have a larger number of samples[0]. Of course, you can't change the effect size, so this would lead to larger necessary samples sizes to study smaller effects.
Be careful with this. Larger sample sizes are more likely to give you a significant result even if one doesn't exist.
Say my null hypothesis is that X=100. The alternative hypothesis is X>100.
What if in reality X is really 100.5? Depending on the problem domain, this may well be the same as the null hypothesis. But a larger sample size is much more likely to give a significant result.
There are ways to fix this, but one should just be aware, though.
- dragonwriter 9y ago> Say my null hypothesis is that X=100. The alternative hypothesis is X>100. > What if in reality X is really 100.5? Then the alternative hypothesis is true and the null is false. (Though normally if the tested hypothesis was X > 100, the null would be X ≤ 100.) > Depending on the problem domain, this may well be the same as the null hypothesis. No, X = 100.5 is not the same as X = 100, irrespective of problem domain, so long as the mathematical symbols have their usual definitions. > But a larger sample size is much more likely to give a significant result. Yes, if something is only just barely true, it is more likely to take a large sample size to distinguish it from the case where it is false. But since in this case, the alternative hypothesis actually is true, it's not a problem that a bigger sample is more likely to reject the null hypothesis.
- BeetleB 9y ago>Then the alternative hypothesis is true and the null is false. (Though normally if the tested hypothesis was X > 100, the null would be X ≤ 100.) Mathematically, yes. Practically, not always. In many real world effects, we're not 100% accurate on what the null hypothesis is. We may think the existing process suggests a the temperature increase of 1C, and so we set that as the null hypothesis. But in that problem domain, we may have difficulty being more precise. Suppose that 1C in the existing process came from the mean of measuring a sample of 100, so the current belief is that the expected value of the increase should be 1C. Suppose in reality it is 1.02C. Now we're proposing a new process and claim it is better than the existing one. This time we take a much larger sample, and see the number hovering at 1.02C - we can then claim the new process is better than the existing process and have a p value to demonstrate it. The math will be correct, and the conclusion will be wrong.
- benchaney 9y ago> Mathematically, yes. Practically, not always. P values measure statistic significance, not practical significance. If you are extrapolating practical significance from statistical significance, then you have a problem that is unrelated to sample size.
- BeetleB 9y ago>P values measure statistic significance, not practical significance. If you are extrapolating practical significance from statistical significance, then you have a problem that is unrelated to sample size If you never extrapolate practical significance from statistical significance, then you have a problem. p values exist for the purpose of doing that. The problem in in research isn't the use of p values per se, but the misuse and misunderstanding of them. An argument to do away with p-values demonstrates a basic misunderstanding of them.
- yorwba 9y agoIf you want to prove that the new process is better on average ("Xnew > Xold") , you would have to use "Xnew <= Xold" as the null. When you determine Xold to be 1C using a sample size of 100, you also get a variance that you have to take into account instead of naively substituting it into the hypothesis. If you do the statistical significance test correctly, you won't be able to show the new process better by taking lots of samples for it, since the variance of the old measurements dominates the uncertainty. (I think, haven't done the math to show it.) If you get wrong conclusions using correct math, your assumptions are wrong. In the case of your example, the incorrect assumption is treating an uncertain measurement as exact when comparing it to another uncertain measurement.
- BeetleB 9y agoI think people are misreading my purpose. Your response comes closest: >If you do the statistical significance test correctly, you won't be able to show the new process better by taking lots of samples for it, since the variance of the old measurements dominates the uncertainty. (I think, haven't done the math to show it.) Higher sample size means lower variance: https://onlinecourses.science.psu.edu/stat414/node/167 https://onlinecourses.science.psu.edu/stat414/node/167
- gweinberg 9y agoNo, if your standard of "statistical significance" is .05, that means you have a 5% of getting a "statistically significant" result.
- BeetleB 9y ago>No, if your standard of "statistical significance" is .05, that means you have a 5% of getting a "statistically significant" result. Your statement didn't address anything in my comment.