3 ms·
That seems to be a pretty sound approach, compared to some of the stuff about multiarmed bandits that shows up here some times. And I certainly expect Noel Wels
by aaronjg 14y ago
That seems to be a pretty sound approach, compared to some of the stuff about multiarmed bandits that shows up here some times. And I certainly expect Noel Welsh to chime in as well.
There are two schools of thought about the approaches to sequential testing, the Bayesian approach lead by Anscombe, and the frequentist by Armitage. I talked a bit about this and outlined Anscombe's approach here [1]. And it is great to see such a nice write up of the frequentist approach and the tables of the stopping criteria
[1] http://blog.custora.com/2012/05/a-bayesian-approach-to-ab-testing/ http://blog.custora.com/2012/05/a-bayesian-approach-to-ab-te...
- btilly 14y agoThanks. If all goes perfectly, I will discuss more ways to think about the problem than just those two, and try to show some connections that may surprise people. But, judging by your nice article, I doubt that I'll prove to have anything to say that you don't already know. :-)
- mwexler 14y agoYou imply that the MAB approach has problems, or at least is not as "sound". Care to state what those problems or issues are?
- aaronjg 14y agoI am mostly critical of claims like '20 lines of code that will beat A/B Testing Every Time.' Multi armed bandits are also not as useful for inference as the frequentist methods that Ben presents in his posts.