There's a related notion of an "adaptive" statistical design, where the allocation of users to each test group varies based on prior performance of the group. F

There's a related notion of an "adaptive" statistical design, where the allocation of users to each test group varies based on prior performance of the group. For example, if after the first 100 users you notice that group A seems to be doing slightly better than group B, you will favor it by allocation more users to that group. You can compute this allocation in such a way as to maximize the number of successes. In particular it will converge to always picking the better approach eventually, assuming there is a real difference. This also means you don't really need to "stop" the experiment to make everyone use the better version (although you may want to for other reasons). Here is one paper: http://web.eecs.umich.edu/~qstout/pap/SciProg00.pdf http://web.eecs.umich.edu/~qstout/pap/SciProg00.pdf