4 ms·
No, no, no, no. https://www.chrisstucchio.com/blog/2015/dont_use_bandits.html https://www.chrisstucchio.com/blog/2015/dont_use_bandits.htm... needs a rebuttal s
by paulddraper 10y ago
No, no, no, no. https://www.chrisstucchio.com/blog/2015/dont_use_bandits.html https://www.chrisstucchio.com/blog/2015/dont_use_bandits.htm... needs a rebuttal so very, very badly.
> Depending on what your website is selling, people will have a different propensity to purchase on Saturday than they have on Tuesday.
Affects multi-armed bandit and fixed tests. If you do fixed A/B test on Tuesday, your results will also be wrong. Either way, you have to decide on what kind of seasonality your data has, and don't make any adjustments until the period is complete.
If anything, multi-armed bandit shines because it can adapt to trends you don't anticipate.
> Delayed response is a big problem when A/B testing the response to an email campaign.
Affects multi-armed bandit and fixed tests. If you include immature data in your p-test, your results will be wrong. Either way, you have to decide how long it takes to declare an individual success or failure.
> You don't get samples for free by counting visits instead of users
Affects multi-armed bandit and fixed tests. Focusing on relevant data increases the power of your experiment.
---
For every single problem, the author admits "A/B tests have the same problem", and then somehow concludes that multi-bandit tests are harder because of these design decisions, despite the fact they affect any experiment process.
- yummyfajitas 10y agoIf anything, multi-armed bandit shines because it can adapt to trends you don't anticipate. It can, but the time it takes is exp(# of samples already passed). You can improve this by using a non-stationary Bayesian model (i.e. one that assumes conversion rates change over time) but this usually involves solving PDEs or something equally difficult. For every single problem, the author admits "A/B tests have the same problem", and then somehow concludes that multi-bandit tests are harder because of these design decisions, despite the fact they affect any experiment process. The point the author (me) is trying to make is not that bandits are fundamentally flawed. The point is that for A/B tests, all these problems have simple fixes: make sure to run the A/B test for long enough. For bandits, the fixes are not nearly as simple. It usually involves non-simple math, or at the very least non-intuitive things (for instance not actually running a bandit until 1 week has passed). At VWO we realized that most of our customers are not sophisticated enough to get all this stuff right, which is why we didn't switch to bandits.
- paulddraper 10y ago> The point is that for A/B tests, all these problems have simple fixes: make sure to run the A/B test for long enough. Multi-bandit has the same fix: make sure the test has run for long enough before adjusting sampling proportions.
- yummyfajitas 10y agoSo what I'm proposing to do is run A/B with a 50/50 split for a full week, then when B wins shift to 0/100 in favor of B. You seem to be proposing to run A/B with a 50/50 split for a full week, then when B does a lot better shift to 10/90 in favor of B and maybe a few weeks later shift to 1/99. What practical benefit do you see to this approach? From my perspective this just slows down the experimental process and keeps losing variations (and associated code complexity) around for a lot longer.
- paulddraper 10y agoFirst, Google Analytics (for example) runs content experiments for a minimum of two weeks regardless of results. It's hardly an unrealistic timeframe for reliable conclusions. > What practical benefit do you see to this approach? Statistically rigorous results, with minimal regret. In your example, you reach the end of the week, and your 50/50 split has one-sided p=0.10, double the usual p<0.05 criteria. What do you do? (a) Call it in favor of B, despite being uncertain about the outcome. (b) Keep running the test. This compromises the statistical rigor of your test. (c) Keep running the test, but use sequential hypothesis testing, e.g. http://elem.com/~btilly/ab-testing-multiple-looks/part1-rigorous.html http://elem.com/~btilly/ab-testing-multiple-looks/part1-rigo... This significantly increases the time to reach a conclusion, and costs you conversions in the meantime. (a) and (b) are the most popular choices, despite them being statistically unjustifiable. http://www.evanmiller.org/how-not-to-run-an-ab-test.html http://www.evanmiller.org/how-not-to-run-an-ab-test.html --- The essential difference when choosing the approach is that 50/50 split optimizes for shortest time to conclusion, and multi-bandit optimizes for fewest failures. In web A/B testing, the latter is usually the most applicable, and for that, you cannot beat Thompson sampling on the average, no matter how diabolically clever your scheme. https://www.lucidchart.com/blog/2016/10/20/the-fatal-flaw-of-ab-tests-peeking/ https://www.lucidchart.com/blog/2016/10/20/the-fatal-flaw-of... There are times when the former is more important, e.g. marketing wants to know how to brand a product that is being released next month. These are the clinical-like experiments that frequentist approaches were formulated for.