4 ms·
> The solution is to use an A/B test running time calculator to determine if you have the required statistical power to run your experiment and for how long you
by mabbo 3y ago
> The solution is to use an A/B test running time calculator to determine if you have the required statistical power to run your experiment and for how long you should run your experiment.
Wouldn't it be better to have an A/B testing system that just counts how many users have been in each assignment group and end when you have the required statistical power?
Time just seems like a stand in for "that should be enough", when in reality you might have a change in how many users get exposed that differs from your expectations.
- aliceryhl 3y agoRunning the experiment until you have a specific pre-determined number of observations is okay. However, the deceptively similar scheme of running it until the results are statistical significant is not okay!
- mreezie 3y agoIf you want statistical significance of 1/20 and you check 20 times... you are likely to find it.