Agree that bayesian approaches would be better here, but disagree that running experiments for preset lengths is a mistake. In a realistic scenario for us, bay

Agree that bayesian approaches would be better here, but disagree that running experiments for preset lengths is a mistake. In a realistic scenario for us, bayesian and frequentist approaches will probably converge to a point that's close enough for a company that runs a website (i.e., we weren't killing anyone by leaving our experiment running). We also weren't getting massive fluctuations in effect size. The cost of the Bayesian approach, in terms of learning a new system of statistics, programming everything up from scratch, and interpreting results, probably wouldn't be worth the efficiency gains. If we were creating an AB testing program, then I would probably do so.