3 ms·
>Then the alternative hypothesis is true and the null is false. (Though normally if the tested hypothesis was X > 100, the null would be X ≤ 100.) Mathematical
by BeetleB 9y ago
>Then the alternative hypothesis is true and the null is false. (Though normally if the tested hypothesis was X > 100, the null would be X ≤ 100.)
Mathematically, yes. Practically, not always.
In many real world effects, we're not 100% accurate on what the null hypothesis is. We may think the existing process suggests a the temperature increase of 1C, and so we set that as the null hypothesis.
But in that problem domain, we may have difficulty being more precise. Suppose that 1C in the existing process came from the mean of measuring a sample of 100, so the current belief is that the expected value of the increase should be 1C. Suppose in reality it is 1.02C. Now we're proposing a new process and claim it is better than the existing one. This time we take a much larger sample, and see the number hovering at 1.02C - we can then claim the new process is better than the existing process and have a p value to demonstrate it.
The math will be correct, and the conclusion will be wrong.
- benchaney 9y ago> Mathematically, yes. Practically, not always. P values measure statistic significance, not practical significance. If you are extrapolating practical significance from statistical significance, then you have a problem that is unrelated to sample size.
- BeetleB 9y ago>P values measure statistic significance, not practical significance. If you are extrapolating practical significance from statistical significance, then you have a problem that is unrelated to sample size If you never extrapolate practical significance from statistical significance, then you have a problem. p values exist for the purpose of doing that. The problem in in research isn't the use of p values per se, but the misuse and misunderstanding of them. An argument to do away with p-values demonstrates a basic misunderstanding of them.
- yorwba 9y agoIf you want to prove that the new process is better on average ("Xnew > Xold") , you would have to use "Xnew <= Xold" as the null. When you determine Xold to be 1C using a sample size of 100, you also get a variance that you have to take into account instead of naively substituting it into the hypothesis. If you do the statistical significance test correctly, you won't be able to show the new process better by taking lots of samples for it, since the variance of the old measurements dominates the uncertainty. (I think, haven't done the math to show it.) If you get wrong conclusions using correct math, your assumptions are wrong. In the case of your example, the incorrect assumption is treating an uncertain measurement as exact when comparing it to another uncertain measurement.
- BeetleB 9y agoI think people are misreading my purpose. Your response comes closest: >If you do the statistical significance test correctly, you won't be able to show the new process better by taking lots of samples for it, since the variance of the old measurements dominates the uncertainty. (I think, haven't done the math to show it.) Higher sample size means lower variance: https://onlinecourses.science.psu.edu/stat414/node/167 https://onlinecourses.science.psu.edu/stat414/node/167
- yorwba 9y ago> Higher sample size means lower variance: That's for a single population. If you are trying to compare two populations, you have two empirical means and two variances. When you do a t-test [1] to compare your measurements, the test statistic has a variance that is approximately computed as variance = variance_1/sample_size_1 + variance_2/sample_size_2 Adding more samples for only one of the populations means that this half of the sum vanishes, but the other half remains unchanged, thus guaranteeing that the variance never drops below a certain threshold. [1] https://en.wikipedia.org/wiki/Student%27s_t-test#Equal_or_unequal_sample_sizes.2C_unequal_variances https://en.wikipedia.org/wiki/Student%27s_t-test#Equal_or_un...
- dragonwriter 9y ago> In many real world effects, we're not 100% accurate on what the null hypothesis is. The null hypothesis is the simple negation of the affirmative hypothesis; anything else and you can't use p-values at all. If you know what the hypothesis you are testing is (and if you don't, you have a fundamental problem), you also know what the null hypothesis is. The proper affirmative and null hypotheses in the test you propose are “process 1 has a greater measure on the dimension of interest than process 0” and “process 1 has no better measure on the dimension of interest than process 0”. Normal statistics on parallel measures of both processes will give you results which may or may not allow you to reject the null hypothesis at a given p-values, but (except that the smallest p-values you can get goes up with sample size, so that you can't, rightly or wrongly, reject the null hypothesis to certain confidence with too small a sample size) the probability of rejecting the null hypothesis incorrectly does not go up with sample size.