5 ms·
A/B Testing vs MAB algorithms - It's complicated
- ckluis 14y agoAt the end of the day, I think A/B testing is utilized more just because it is easier to understand & implement. I would love to utilize something like unbounce for MAB testing, but I haven't seen good MAB frameworks that are ready for the average marketer to use. It would be nice to have a MAB framework that utilizes MVT so that you could preload 100s of alternates (c2a, headline, copy, images, placement) and see let it go to town.
- larrydag 14y agoGood fair analysis of both implementations. I really agree with the article writer that the best implementation is the one that is right for their business.
- btilly 14y agoMy hope is that I helped people figure out which one is right for their business. :-)
- btilly 14y agoYou make good points. However the ultra-simple MAB for home pages is a solvable problem. Noel at Myna has been responsive to me. If you give him feedback on how he can target that market segment, particularly if you're willing to be a paying customer, he'll likely be responsive. MVT is inherently more complex. I guarantee you that there are MAB based systems that do MVT on problems of insane complexity. I pointed at one system at Yahoo that likely is an example. And the potential win isn't small. Yahoo claims a 300% increase in clicks on their home page module since implementing it. But I haven't noticed anything like that publicly available. (I have seen companies that do it behind the scenes - but the one that first comes to mind hides that detail and tries to be priced on a pay for performance model.) Of course a company that already does MAB could build a custom solution. Or you can try to build your own if you think there is money in it. My personal life does not currently permit me the time commitment that a startup would require. But if you or someone else wants to build it, I'd be willing to consult on how to make it work.
- ckluis 14y agoMy first piece of advice is that screenshots & videos are good. Picture is 1,000 words. A video is 30,000 words a second. I dream of a MAB/MVT combo solution. I have a project coming up that I might try this out on.
- noelwelsh 14y agoUsability for non-programmers is certainly not a strong point of Myna at the moment. Right now the best place to look is the demos: http://myna.github.com/myna-js/ http://myna.github.com/myna-js/ The redirection demo is an example of how you can do something like Unbounce. In terms of the MVT style stuff you discuss, we're in very early stages of looking at it. Not sure when it'll arrive -- depends on how much interest we get.
- cmansley 14y agoI think the real distinction here is stationary verses non-stationary distributions. Many of the arguments made in this article hinge on the fact that the responses to the same input change over time (nights are different then daytime, which is different than the weekends). By continuously running the A/B testing, you are looking at a small window in time, which you assume is stationary so you can do your t-tests or whatever statistical test. But, to be clear, this is a heuristic in A/B testing. If you have a window of time over which you know your distributions are stationary, you should always use a logarithmic regret MAB algorithm because it is theoretically better. I think the best way to frame this argument is that because the domain does not match the assumptions of MAB, A/B testing has been shown to be robust and easy to modify for non-stationary domains, while logarithmic regret algorithms are somewhat more fragile.
- btilly 14y agoCorrection. A/B testing does not need to assume that it is testing over a small window in time that has stationary conversion rates. In fact in practice tests run for a long window of time over which you have good evidence that the distribution is not stationary. For example in the middle of running a long test that eventually found a small lift, I've often run and rolled out a second test that generated a much stronger lift. The weaker assumption that you need is that the preference between versions is stable across your fluctuations. Then because the mix of versions is time independent, that non-stationary fluctuation is not statistically different between the slice that was put into A and the slice that was put into B. And therefore the variation between different samples becomes just another unknown random factor that does not interfere with your statistical analysis of whether there is a difference. This is an advantage between A/B testing and the current state of the art in MAB algorithms. When this came up before, Noel (at Myna) and I privately did an admittedly brief search of the literature for discussion of this point with regards to MAB algorithms. We turned up a number of things that would work in the long run, but none that directly addressed the problem. But in discussion we did manage to come up with effective MAB algorithms, whose regret is only a constant factor worse than standard MAB algorithms, that accurately will identify stable preferences in the face of constantly fluctuating conversion rates. To the best of my knowledge nobody, including Noel, has yet implemented such algorithms in practice. But in principle it can be done. However even if you do it, several of my other points still apply as real differences.
- adrianhoward 14y agoI thank you from the bottom of my heart for writing this - since I can now cross of the "Write something comparing bandit & a/b" item from my to do list and point people to your post instead :-)
- deleted 14y ago[deleted]
- btilly 14y agoThe win from actually testing is ridiculously more important than the marginal differences between different ways of running tests. Even if you later find out that your testing methodology was wrong, it is still a likely win. If you've not already been convinced of that, there is no more important point that I can make to you about testing. And, sadly, most companies out there do not seem to have realized this point yet. But continually repeating it is boring for me, and useless for people who've already got that point. That said, I would add a third case to your list where MAB is helpful. If you're clever with a MAB system, you can manage to roll new versions into the testing mix without ruining an ongoing test that you don't yet have final statistics on. This flexibility can be very helpful.
- deleted 14y ago[deleted]
- patio11 14y agoI really want to tell you all to read this. The reality is (as it points out in the text) the overwhelming majority of you don't A/B test. If you don't A/B test already you should stop reading about A/B testing and start A/B testing. It will be the most important thing you do today. It will probably be the most important single thing you do in quite a while. For the overwhelming majority of people A/B testing, this is similarly not really a priority for you today. You're fine doing what you're doing. (But... test more.) But for the sliver of a sliver of people who really need to care about moving the state of the art in conversion optimization forward, this essay is very important. If I had an A/BatSignal I would be shining it on this.
- paraschopra 14y agoThanks for writing this Ben. It's a very fair and balanced assessment of both techniques.
- psb217 14y agoOver the last few years, some people have started to work on a problem that has been described as "pure exploration" in multi-armed bandits [1,2]. The objective in this problem is, roughly speaking, to maximize the rate at which you become certain about whether or not you've correctly identified the best bandit arm (or, more generally, the top n arms). This solves some of the more common complaints about classic MAB algorithms in that, when run for a finite number of trials, the resulting sampling of the arms produces a far more confident decision than either the regret-minimizing policies followed by standard MAB algorithms or the uniform allocation policies typically used in A/B testing. The original article mentions problems in MAB algorithms dealing with delayed feedback. Such issues are largely ameliorated by the use of algorithms related to "Thompson sampling" [3], which induces stochastic trial allocation policies rather than the deterministic policies induced by UCB's selection process. It's definitely possible to develop Thompson-like methods for the exploration-oriented MAB problem, and such methods can rapidly distinguish the best among a rather large set of options, as might be required in applications like MVT (see note below). Note: I'm currently doing academic research in this area and, if anyone's particularly interested, I could share some of the (simulated) empirical data and algorithmic details pertinent to what I said above. [1]: Multi-Bandit Best Arm Identification, Gabillon et. al, NIPS 2011 [2]: PAC Subset Selection in Multi-Armed Bandits, Kalyanakrishnan and Stone, ICML 2012 [3]: An Empirical Evaluation of Thompson Sampling, Chappelle and Li, NIPS 2011