6 ms·

This is yet another article that ignores the fact that there is a MUCH better approach to this problem. Thompson sampling avoids the problems of multiple testi

This is yet another article that ignores the fact that there is a MUCH better approach to this problem. Thompson sampling avoids the problems of multiple testing, power, early stopping and so on by starting with a proper Bayesian approach. The idea is that the question we want to answer is more "Which alternative is nearly as good as the best with pretty high probability?". This is very different from the question being answered by a classical test of significance. Moreover, it would be good if we could answer the question partially by decreasing the number of times we sample options that are clearly worse than the best. What we want to solve is the multi-armed bandit problem, not the retrospective analysis of experimental results problem. The really good news is that Thompson sampling is both much simpler than hypothesis testing can be done in far more complex situations. It is known to be an asymptotically optimal solution to the multi-armed bandit and often takes only a few lines of very simple code to implement. See http://tdunning.blogspot.com/2012/02/bayesian-bandits.html http://tdunning.blogspot.com/2012/02/bayesian-bandits.html for an essay and see https://github.com/tdunning/bandit-ranking https://github.com/tdunning/bandit-ranking for an example applied to ranking.

mattj 10y ago
I agree with you (and love your blog, btw), but I think you're skipping over at least a few benefits you can get out of a mature / well built a/b framework that are hard to build into a bandit approach. The biggest one I've found personally useful is days-in analysis; for example, quantifying the impact of a signup-time experiment on one-week retention. This doesn't really apply to learning ranking functions or other transactional (short-feedback loop) optimization. That being said, building a "proper" a/b harness is really hard and will be a constant source of bugs / FUD around decision-making (don't believe me? try running an a/a experiment and see how many false positives you get). I've personally built a dead-simple bandit system when starting greenfield and would recommend the same to anyone else.
- paulddraper 10y ago
  Speaking of mature, well-built A/B test frameworks, Google Analytics uses multi-armed bandit. https://support.google.com/analytics/answer/2844870?hl=en https://support.google.com/analytics/answer/2844870?hl=en
  - mhoad 10y ago
    Probably worth mentioning that the Google Content Experiments framework is in the process of being replaced with Google Optimize (currently in a private beta) which does NOT make use of multi-armed bandits much to my confusion and disappointment.
    - paulddraper 10y ago
      Huh. So do you know if they do anything help with repeat testing/peeking? Optimizely takes an interesting approach: they apply repeat testing methods, segmenting the tests by user views of the results. Like 30x more complicated than multi-bandit, but they don't need a feedback mechanism.
developer2 10y ago
This ignores what I have seen in my experience, which is that marketing teams - composed of the people who dictate what A/B tests the business should run - have little to no background in statistics, let alone any interest whatsoever in actually performing legitimate A/B tests. It's often the case that the decision maker has already decided to move ahead with option A, but performs a minimal "fake" A/B test to put in their report as a way to justify their choice. I've seen A/B tests deployed at 10am in the morning, and taken down at 1pm with less than a dozen data points collected. The A/B test "owner" is happy to see that option A resulted in 7 conversions, with option B only having 5. Not statistically significant whatsoever, but hey let's waste developers' time and energy for two days implementing an A/B test in order to help someone else try to nab their quarterly marketing bonus.
- paulddraper 10y ago
  Join us, comrade, in the fight against the statistical blight! Move your decision process to multi-armed bandit and you never have to decide when to end an A/B test -- math does it for you, in a provably optimal way.
  - RA_Fisher 10y ago
    I'm not sure this solves it because you have to have a really strong sense of the loss function to pull it off. That's much easier to intuit and use to guide experiments than actually build into the bandit algo.
    - paulddraper 10y ago
      > That's much easier to intuit and use to guide experiments than actually build into the bandit algo. IDK about your intuition, but for most other people, it gets in the way of statistics. The "loss function" is just as easy to calculate for A/B tests as for multi-armed bandit. The value of user doing A is $X, the value of B is $Y, and the value of C is $Z.
      - 10y ago
paulddraper 10y ago
+1 I wrote a blog post focusing on the #2 problem mentioned: multiple testing. I simulated the typical mitigating approaches, and comparing them against Thompson sampling (code in GitHub). https://www.lucidchart.com/blog/2016/10/20/the-fatal-flaw-of-ab-tests-peeking/ https://www.lucidchart.com/blog/2016/10/20/the-fatal-flaw-of... I'm not a Bayesian fanatic, but given how perfectly A/B test optimization fits in the Bayesian approach, it's a shame it's not yet the de facto standard. I think the primary reasons are (1) it's not as intuitive (especially for the uninitiated), (2) it's harder to implement an automated feedback mechanism, (3) FUD. E.g. https://www.chrisstucchio.com/blog/2015/dont_use_bandits.html https://www.chrisstucchio.com/blog/2015/dont_use_bandits.htm... lists devastating complications with correct multi-armed bandit tests, and then in fine print admits that traditional tests have all the same complications.
- amasad 10y ago
  This is a great read. But left me thinking I'm missing in the fundamentals. Can you recommend a book (or other posts) on statistics fundementals for programmers?
  - abecedarius 10y ago
    If no one actually knowledgeable answers, I'll say I liked what I read of Doing Bayesian Data Analysis -- early partial draft here: https://faculty.washington.edu/jmiyamot/p548/kruschkejk%20bayesian%20data%20analysis.ch01-06.Pdf https://faculty.washington.edu/jmiyamot/p548/kruschkejk%20ba... I didn't persevere because the R language rubs me the wrong way and I'd be comfier with Python or Haskell or something. http://camdavidsonpilon.github.io/Probabilistic-Programming-and-Bayesian-Methods-for-Hackers/ http://camdavidsonpilon.github.io/Probabilistic-Programming-... looks promising in that vein, but I haven't read it.
- saycheese 10y ago
  Given most people don't get A/B testing, it's a stretch to me to believe that some that does would know about more complex approaches that require more skill. Firmly believe that there's way more to be gained by more people understanding how to use A/B testing than more complex solutions.
tedsanders 10y ago
Thank you Ted for bringing sanity to this conversation. Terrific point and post. By the way, I doubt you remember me, but thank you for inviting me on a tour of Veoh ten years ago when I was a young college sophomore. I enjoyed the opportunity as well as our brief chat about Bayesianism.
- ted_dunning 10y ago
  My calendar agrees with you ... we apparently had lunch in December, 2007. Sadly, I don't remember it off-hand. On the other hand, one reason is that I invited a lot of students to come visit (and a number to intern with us).
yummyfajitas 10y ago
Thompson sampling is a great tool. I've used it to make reasonably large amounts of money. But it does not solve the same problem as A/B testing. Thompson Sampling (at least the standard approach) assumes that conversion rates do not change. In reality they vary significantly over a week, and this fundamentally breaks bandit algorithms. https://www.chrisstucchio.com/blog/2015/dont_use_bandits.html https://www.chrisstucchio.com/blog/2015/dont_use_bandits.htm... Furthermore, you do not need to use Thompson Sampling to have a proper Bayesian approach. At VWO we also use a proper Bayesian approach, but we use A/B testing in to avoid the various pitfalls that Thompson Sampling has. Google Optimize uses an approach very similar to ours, (although it may be flawed [1]) and so does A/B Tasty (probably not flawed). https://cdn2.hubspot.net/hubfs/310840/VWO_SmartStats_technical_whitepaper.pdf https://cdn2.hubspot.net/hubfs/310840/VWO_SmartStats_technic... Note: I'm the Director of Data Science at VWO. Obviously I'm biased, etc. However my post critiquing bandits was published before I took on this role. It was a followup to a previous post of mine which led people to accidentally misuse bandits: https://www.chrisstucchio.com/blog/2012/bandit_algorithms_vs_ab.html https://www.chrisstucchio.com/blog/2012/bandit_algorithms_vs... [1] The head of data science at A/B Tasty suggests Google Optimize counts sessions rather than visitors, which would break the IID assumption. https://www.abtasty.com/uk/blog/data-scientist-hubert-google-optimize/ https://www.abtasty.com/uk/blog/data-scientist-hubert-google...
- paulddraper 10y ago
  No, no, no, no. https://www.chrisstucchio.com/blog/2015/dont_use_bandits.html https://www.chrisstucchio.com/blog/2015/dont_use_bandits.htm... needs a rebuttal so very, very badly. > Depending on what your website is selling, people will have a different propensity to purchase on Saturday than they have on Tuesday. Affects multi-armed bandit and fixed tests. If you do fixed A/B test on Tuesday, your results will also be wrong. Either way, you have to decide on what kind of seasonality your data has, and don't make any adjustments until the period is complete. If anything, multi-armed bandit shines because it can adapt to trends you don't anticipate. > Delayed response is a big problem when A/B testing the response to an email campaign. Affects multi-armed bandit and fixed tests. If you include immature data in your p-test, your results will be wrong. Either way, you have to decide how long it takes to declare an individual success or failure. > You don't get samples for free by counting visits instead of users Affects multi-armed bandit and fixed tests. Focusing on relevant data increases the power of your experiment. --- For every single problem, the author admits "A/B tests have the same problem", and then somehow concludes that multi-bandit tests are harder because of these design decisions, despite the fact they affect any experiment process.
martingoodson 10y ago
This is yet another comment claiming that Thompson sampling is the answer to all of our statistical problems! Naive Thompson sampling (like the code you linked to) will result in problems equally disastrous to those I wrote about in the Qubit whitepaper. Other comments have highlighted a key problem with simple bandit algorithms - reward distributions which change over time will render their results worthless. You can model these dynamics but not in 'a few lines of very simple code'. It is verging on the irresponsible to suggest otherwise. I personally favour a bayesian state-space model to elegantly take care of these things - but that's outside the remit of the whitepaper and outside the skill set of most non-statisticians. Frequentist testing, when done properly, is simple to implement and has statistical guarantees that are very attractive in practice.

mattj 10y ago

I agree with you (and love your blog, btw), but I think you're skipping over at least a few benefits you can get out of a mature / well built a/b framework that are hard to build into a bandit approach. The biggest one I've found personally useful is days-in analysis; for example, quantifying the impact of a signup-time experiment on one-week retention. This doesn't really apply to learning ranking functions or other transactional (short-feedback loop) optimization. That being said, building a "proper" a/b harness is really hard and will be a constant source of bugs / FUD around decision-making (don't believe me? try running an a/a experiment and see how many false positives you get). I've personally built a dead-simple bandit system when starting greenfield and would recommend the same to anyone else.

paulddraper 10y ago
Speaking of mature, well-built A/B test frameworks, Google Analytics uses multi-armed bandit. https://support.google.com/analytics/answer/2844870?hl=en https://support.google.com/analytics/answer/2844870?hl=en
- mhoad 10y ago
  Probably worth mentioning that the Google Content Experiments framework is in the process of being replaced with Google Optimize (currently in a private beta) which does NOT make use of multi-armed bandits much to my confusion and disappointment.
  - paulddraper 10y ago
    Huh. So do you know if they do anything help with repeat testing/peeking? Optimizely takes an interesting approach: they apply repeat testing methods, segmenting the tests by user views of the results. Like 30x more complicated than multi-bandit, but they don't need a feedback mechanism.

developer2 10y ago

This ignores what I have seen in my experience, which is that marketing teams - composed of the people who dictate what A/B tests the business should run - have little to no background in statistics, let alone any interest whatsoever in actually performing legitimate A/B tests. It's often the case that the decision maker has already decided to move ahead with option A, but performs a minimal "fake" A/B test to put in their report as a way to justify their choice. I've seen A/B tests deployed at 10am in the morning, and taken down at 1pm with less than a dozen data points collected. The A/B test "owner" is happy to see that option A resulted in 7 conversions, with option B only having 5. Not statistically significant whatsoever, but hey let's waste developers' time and energy for two days implementing an A/B test in order to help someone else try to nab their quarterly marketing bonus.

paulddraper 10y ago
Join us, comrade, in the fight against the statistical blight! Move your decision process to multi-armed bandit and you never have to decide when to end an A/B test -- math does it for you, in a provably optimal way.
- RA_Fisher 10y ago
  I'm not sure this solves it because you have to have a really strong sense of the loss function to pull it off. That's much easier to intuit and use to guide experiments than actually build into the bandit algo.
  - paulddraper 10y ago
    > That's much easier to intuit and use to guide experiments than actually build into the bandit algo. IDK about your intuition, but for most other people, it gets in the way of statistics. The "loss function" is just as easy to calculate for A/B tests as for multi-armed bandit. The value of user doing A is $X, the value of B is $Y, and the value of C is $Z.
    - 10y ago

paulddraper 10y ago

+1 I wrote a blog post focusing on the #2 problem mentioned: multiple testing. I simulated the typical mitigating approaches, and comparing them against Thompson sampling (code in GitHub). https://www.lucidchart.com/blog/2016/10/20/the-fatal-flaw-of-ab-tests-peeking/ https://www.lucidchart.com/blog/2016/10/20/the-fatal-flaw-of... I'm not a Bayesian fanatic, but given how perfectly A/B test optimization fits in the Bayesian approach, it's a shame it's not yet the de facto standard. I think the primary reasons are (1) it's not as intuitive (especially for the uninitiated), (2) it's harder to implement an automated feedback mechanism, (3) FUD. E.g. https://www.chrisstucchio.com/blog/2015/dont_use_bandits.html https://www.chrisstucchio.com/blog/2015/dont_use_bandits.htm... lists devastating complications with correct multi-armed bandit tests, and then in fine print admits that traditional tests have all the same complications.

amasad 10y ago
This is a great read. But left me thinking I'm missing in the fundamentals. Can you recommend a book (or other posts) on statistics fundementals for programmers?
- abecedarius 10y ago
  If no one actually knowledgeable answers, I'll say I liked what I read of Doing Bayesian Data Analysis -- early partial draft here: https://faculty.washington.edu/jmiyamot/p548/kruschkejk%20bayesian%20data%20analysis.ch01-06.Pdf https://faculty.washington.edu/jmiyamot/p548/kruschkejk%20ba... I didn't persevere because the R language rubs me the wrong way and I'd be comfier with Python or Haskell or something. http://camdavidsonpilon.github.io/Probabilistic-Programming-and-Bayesian-Methods-for-Hackers/ http://camdavidsonpilon.github.io/Probabilistic-Programming-... looks promising in that vein, but I haven't read it.
saycheese 10y ago
Given most people don't get A/B testing, it's a stretch to me to believe that some that does would know about more complex approaches that require more skill. Firmly believe that there's way more to be gained by more people understanding how to use A/B testing than more complex solutions.

tedsanders 10y ago

Thank you Ted for bringing sanity to this conversation. Terrific point and post. By the way, I doubt you remember me, but thank you for inviting me on a tour of Veoh ten years ago when I was a young college sophomore. I enjoyed the opportunity as well as our brief chat about Bayesianism.

ted_dunning 10y ago
My calendar agrees with you ... we apparently had lunch in December, 2007. Sadly, I don't remember it off-hand. On the other hand, one reason is that I invited a lot of students to come visit (and a number to intern with us).

yummyfajitas 10y ago

Thompson sampling is a great tool. I've used it to make reasonably large amounts of money. But it does not solve the same problem as A/B testing. Thompson Sampling (at least the standard approach) assumes that conversion rates do not change. In reality they vary significantly over a week, and this fundamentally breaks bandit algorithms. https://www.chrisstucchio.com/blog/2015/dont_use_bandits.html https://www.chrisstucchio.com/blog/2015/dont_use_bandits.htm... Furthermore, you do not need to use Thompson Sampling to have a proper Bayesian approach. At VWO we also use a proper Bayesian approach, but we use A/B testing in to avoid the various pitfalls that Thompson Sampling has. Google Optimize uses an approach very similar to ours, (although it may be flawed [1]) and so does A/B Tasty (probably not flawed). https://cdn2.hubspot.net/hubfs/310840/VWO_SmartStats_technical_whitepaper.pdf https://cdn2.hubspot.net/hubfs/310840/VWO_SmartStats_technic... Note: I'm the Director of Data Science at VWO. Obviously I'm biased, etc. However my post critiquing bandits was published before I took on this role. It was a followup to a previous post of mine which led people to accidentally misuse bandits: https://www.chrisstucchio.com/blog/2012/bandit_algorithms_vs_ab.html https://www.chrisstucchio.com/blog/2012/bandit_algorithms_vs... [1] The head of data science at A/B Tasty suggests Google Optimize counts sessions rather than visitors, which would break the IID assumption. https://www.abtasty.com/uk/blog/data-scientist-hubert-google-optimize/ https://www.abtasty.com/uk/blog/data-scientist-hubert-google...

paulddraper 10y ago
No, no, no, no. https://www.chrisstucchio.com/blog/2015/dont_use_bandits.html https://www.chrisstucchio.com/blog/2015/dont_use_bandits.htm... needs a rebuttal so very, very badly. > Depending on what your website is selling, people will have a different propensity to purchase on Saturday than they have on Tuesday. Affects multi-armed bandit and fixed tests. If you do fixed A/B test on Tuesday, your results will also be wrong. Either way, you have to decide on what kind of seasonality your data has, and don't make any adjustments until the period is complete. If anything, multi-armed bandit shines because it can adapt to trends you don't anticipate. > Delayed response is a big problem when A/B testing the response to an email campaign. Affects multi-armed bandit and fixed tests. If you include immature data in your p-test, your results will be wrong. Either way, you have to decide how long it takes to declare an individual success or failure. > You don't get samples for free by counting visits instead of users Affects multi-armed bandit and fixed tests. Focusing on relevant data increases the power of your experiment. --- For every single problem, the author admits "A/B tests have the same problem", and then somehow concludes that multi-bandit tests are harder because of these design decisions, despite the fact they affect any experiment process.

martingoodson 10y ago

This is yet another comment claiming that Thompson sampling is the answer to all of our statistical problems! Naive Thompson sampling (like the code you linked to) will result in problems equally disastrous to those I wrote about in the Qubit whitepaper. Other comments have highlighted a key problem with simple bandit algorithms - reward distributions which change over time will render their results worthless. You can model these dynamics but not in 'a few lines of very simple code'. It is verging on the irresponsible to suggest otherwise. I personally favour a bayesian state-space model to elegantly take care of these things - but that's outside the remit of the whitepaper and outside the skill set of most non-statisticians. Frequentist testing, when done properly, is simple to implement and has statistical guarantees that are very attractive in practice.