3 ms·
The point is that a larger sample size increases precision: the confidence you get rejecting a hypothesis depends on that and the size of the effect. If I tell
by rcxdude 2y ago
The point is that a larger sample size increases precision: the confidence you get rejecting a hypothesis depends on that and the size of the effect. If I tell you I've sampled a process and gotten the values -2, 0, and 15, you can reject the hypothesis that the process is a gaussian process with mean 10000 and standard deviation 0.1 with extreme confidence. Sampling further will almost certainly not change that (in fact you would need an impractical number of samples in this extreme case).
The reason you need to tighten your bounds for an early stop is to deal with the fact that you are effectively running multiple experiments: it's possible by chance that you are stopping at a point where the data seems to show an effect by random variation, but because you are not repeating all the samples so far, there's a lot fewer possible outcomes, so the tightening of the bounds that you need is not so much. (p-hacking is simply not applying the appropriate correction for this).