3 ms·
Notably absent from this article is any discussion about multi-armed bandits. A/B testing only leads to finding the more optimal of treatments, where as multi-a
by jameslk 8y ago
Notably absent from this article is any discussion about multi-armed bandits. A/B testing only leads to finding the more optimal of treatments, where as multi-armed bandit algorithms help find the optimal treatment and exploit it during testing. If profit is involved, it seems obvious that you should be considering both exploring treatments but most importantly exploiting the treatments that currently yields the most profit.
Google has a pretty good FAQ on this:
https://support.google.com/analytics/answer/2847021?hl=en&ref_topic=1745207 https://support.google.com/analytics/answer/2847021?hl=en&re...
- paulddraper 8y agoYep, and GA does multi armed bandit testing. Definitely the way to go for A/B.
- joshuamorton 8y agoGonna toot my own horn here for a second, but here[0] is a presentation I did on multi-armed bandits (and specifically Thompson sampling, the tragically underutilized optimal bandit method) to an undergrad ML group. Amusingly, the diagram I use on slide 3 is from a research paper from Microsoft. We had a guest speaker from MS's exp-platform team [1] the prior week, and she had discussed A/B testing, but not touched on bandits, and I felt a need to make this exact point. [0]: https://gtagency.github.io/2016/experimentation-with-no-ragrets https://gtagency.github.io/2016/experimentation-with-no-ragr... [1]: https://exp-platform.com/ https://exp-platform.com/