2 ms·
Bandits and A/B are meant for solving very different problem. In bandit there is a clear explore/exploit trade off. There is no such trade off in the A/B formu
by srean 8y ago
Bandits and A/B are meant for solving very different problem.
In bandit there is a clear explore/exploit trade off. There is no such trade off in the A/B formulation, although it does get used in scenarios that have such trade offs.
If I can pull the lever a finite and small number of times there is a strong incentive for using a bandit. In this case I don't want to pull the wrong lever as few times as possible. On the other hand if I am given an unlimited number of pulls, I can afford to pull the wrong one many times more (still finite) for the sake of 'knowledge' knowing well that I would have infinitely many opportunities to exploit that knowledge.
- joshuamorton 8y agoAnd for a business there always is. In an a/b test that is to improve customer conversion, in a perfect world, you use the superior method on everyone, converting the maximum number of customers. That saves you money. In other words, the opportunity cost of putting someone in the wrong group creates such a trade-off. You can pull the lever as many times as you want, but each one potentially costs you money. It's textbook bandits.
- srean 8y agoTo a large extent I agree with you. Differences creep in when there is ambiguity and judgement involved on what is that metric that the org wants to optimize. This is fairly common. Typically, in these situations its the PMs who make the final call. There the goal of the experiment protocol is glean as much knowledge as possible, and present it to the PM. The thinking there is -- if that comes at the cost of exposing some customers to bad choices, so be it.