This is exactly the test you are looking for: http://www.bayesianwitch.com/blog/2014/bayesian_ab_test.html http://www.bayesianwitch.com/blog/2014/bayesian_ab_t

This is exactly the test you are looking for: http://www.bayesianwitch.com/blog/2014/bayesian_ab_test.html http://www.bayesianwitch.com/blog/2014/bayesian_ab_test.html It can be drastically improved in the case of emails by building a Bayesian model tuned to email itself. Most people just blindly apply a testing procedure to their situation, but if you tune the test to your situation you can make do with a LOT fewer samples.