2 ms·
> Is the author saying one must wait for 6000 events even if the difference between A/B is large? Yes. > The larger the relative difference, the fewer events
by user5994461 10y ago
> Is the author saying one must wait for 6000 events even if the difference between A/B is large?
Yes.
> The larger the relative difference, the fewer events needed.
Wrong. That's the common mistake that is being addressed.
The relative difference could be anything at any point in time. You always need a lots of events to be somewhat confident that there is really a difference.
Short version: If you want X% confidence on a test, there are maths formula to say that you need to monitor N events, at least. The formula doesn't depends on how large is the difference (which you don't know anyway).
- tedsanders 10y agoIt's absolutely true that if the difference is larger you need fewer samples to reach statistical significance. This why is why small effects require studies with large sample sizes.
- contravariant 10y agoWell, in this case the distribution is (assumed to be) binomial, which does allow you to assign a higher significance for cases where there is a strong effect.