3 ms·

Ravi Parikh from Heap goes into detail about why early stopping can screw up your test results: http://data.heapanalytics.com/dont-stop-your-ab-tests-part-way-t

Ravi Parikh from Heap goes into detail about why early stopping can screw up your test results: http://data.heapanalytics.com/dont-stop-your-ab-tests-part-way-through/ http://data.heapanalytics.com/dont-stop-your-ab-tests-part-w...

beejiu 12y ago
Well, testing to a fixed sample to run your test and then sending to the other 80% is not an example of early stopping. If you want to send 8,000 emails and conduct a test on the first 1,000, then that is not early stopping. It would be early stopping if you looked at the preliminary results and decided to stop at 500 instead. Otherwise, the statistical interpretation is sound. There are statistical methods to deal with early stopping if you don't want to complete tests on huge samples (e.g. Sequential Probability Ratio Tests), which may be suitable in some cases.
- noelwelsh 12y ago
  With email you don't know if the user isn't going to open your email, or they just haven't opened it yet. So there is a survivor analysis aspects to A/B testing of email -- at any point in time you might get more information from a fixed sample just by waiting. This definitely effects the power of tests. However there is an early stopping aspect in the decision to wait longer to collect more information, or to stop the test. This is analogous to deciding to collect more samples or stop a test, which is classic early stopping. In both cases you're making a decision to stop/continue without considering its impact on your error rate. On closer reading I note that MailChimp by default avoids this issue, by making a decision after a fixed period of time. However they aren't controlling for loss of power and so forth. ----- Other points: - With a finite population, such as a mailing list, minimising regret is a better objective than statistical significance. (Want to know more? Sign up at bandits.mynaweb.com and it will be covered in due course.) - There is a multiple testing aspect to what MailChimp does, and I bet they don't control for false discovery rate. - Also some post-hoc reasoning. - Statistics is hard, and it's easy to criticise others and hard to do well oneself. I hope this post will be taken as constructive criticism.

beejiu 12y ago

Well, testing to a fixed sample to run your test and then sending to the other 80% is not an example of early stopping. If you want to send 8,000 emails and conduct a test on the first 1,000, then that is not early stopping. It would be early stopping if you looked at the preliminary results and decided to stop at 500 instead. Otherwise, the statistical interpretation is sound. There are statistical methods to deal with early stopping if you don't want to complete tests on huge samples (e.g. Sequential Probability Ratio Tests), which may be suitable in some cases.

noelwelsh 12y ago
With email you don't know if the user isn't going to open your email, or they just haven't opened it yet. So there is a survivor analysis aspects to A/B testing of email -- at any point in time you might get more information from a fixed sample just by waiting. This definitely effects the power of tests. However there is an early stopping aspect in the decision to wait longer to collect more information, or to stop the test. This is analogous to deciding to collect more samples or stop a test, which is classic early stopping. In both cases you're making a decision to stop/continue without considering its impact on your error rate. On closer reading I note that MailChimp by default avoids this issue, by making a decision after a fixed period of time. However they aren't controlling for loss of power and so forth. ----- Other points: - With a finite population, such as a mailing list, minimising regret is a better objective than statistical significance. (Want to know more? Sign up at bandits.mynaweb.com and it will be covered in due course.) - There is a multiple testing aspect to what MailChimp does, and I bet they don't control for false discovery rate. - Also some post-hoc reasoning. - Statistics is hard, and it's easy to criticise others and hard to do well oneself. I hope this post will be taken as constructive criticism.