With email you don't know if the user isn't going to open your email, or they just haven't opened it yet. So there is a survivor analysis aspects to A/B testing

With email you don't know if the user isn't going to open your email, or they just haven't opened it yet. So there is a survivor analysis aspects to A/B testing of email -- at any point in time you might get more information from a fixed sample just by waiting. This definitely effects the power of tests. However there is an early stopping aspect in the decision to wait longer to collect more information, or to stop the test. This is analogous to deciding to collect more samples or stop a test, which is classic early stopping. In both cases you're making a decision to stop/continue without considering its impact on your error rate. On closer reading I note that MailChimp by default avoids this issue, by making a decision after a fixed period of time. However they aren't controlling for loss of power and so forth. ----- Other points: - With a finite population, such as a mailing list, minimising regret is a better objective than statistical significance. (Want to know more? Sign up at bandits.mynaweb.com and it will be covered in due course.) - There is a multiple testing aspect to what MailChimp does, and I bet they don't control for false discovery rate. - Also some post-hoc reasoning. - Statistics is hard, and it's easy to criticise others and hard to do well oneself. I hope this post will be taken as constructive criticism.