[I'm a PM @ Optimizely] We were asked about this article before, on our community forums, and one of our statisticians, David, wrote a detailed reply to this a

[I'm a PM @ Optimizely] We were asked about this article before, on our community forums, and one of our statisticians, David, wrote a detailed reply to this article's concerns about one- vs two-tailed testing, which might be of interest [3rd from the top]: https://community.optimizely.com/t5/Strategy-Culture/Let-s-talk-about-Single-Tailed-vs-Double-Tailed/m-p/4220 https://community.optimizely.com/t5/Strategy-Culture/Let-s-t... Additionally, since then, as other commenters have mentioned, we've completely overhauled how we do our A/B-testing calculations, which, theoretically and empirically, now have an accurate false-positive rate even when monitored continuously. Details: https://blog.optimizely.com/2015/01/20/statistics-for-the-internet-age-the-story-behind-optimizelys-new-stats-engine/ https://blog.optimizely.com/2015/01/20/statistics-for-the-in...