4 ms·
Decide your sample size beforehand, and only stop the test after you've reached it. That may be the gospel among A/B testers, but from a Bayesian perspective,
by ced 16y ago
Decide your sample size beforehand, and only stop the test after you've reached it.
That may be the gospel among A/B testers, but from a Bayesian perspective, this is definitely not the best you can do (though it won't necessarily be wrong). Beware, Bayesian methods are provably optimal.
The details are here for the interested (chap. 4 & 6):
http://www-biba.inrialpes.fr/Jaynes/prob.html http://www-biba.inrialpes.fr/Jaynes/prob.html
It's not casual reading... I wish there was something shorter available.
- StavrosK 16y agoThank you, I downloaded the Bayesian method paper linked in the article and was going to read it, I'm very interested in a probably optimal stopping method (obviously). I'm only afraid it'll fly over my head, despite my having a bit of stats background. Thanks again!
- ced 16y agoYou mean this paper? Berry, Donald A. “Bayesian Statistics and the Efficiency and Ethics of Clinical Trials,” Statistical Science, Vol. 19, No. 1 (Feb., 2004), pp. 175-187 I haven't heard of it. But if you're seriously motivated to do things right and you're "mathematically mature", I would advise you to read Jaynes' book linked above. For me it was a huge eye opener. The screed of the Bayesians is "You don't need a separate statistics field! You just need the laws of probabilities, and that's enough to answer any question you want." Let me know if you need help, my email is in my profile.
- StavrosK 16y agoYep, that's the paper. I'll try to read the book when I find some time (thankfully, "get more data" is good enough for now). Thanks again for your help.