3 ms·
> The purpose of an A/B test isn't to always show the best performing result, it's to perform a _controlled scientific experiment_ with a control group, from wh
by paulddraper 7y ago
> The purpose of an A/B test isn't to always show the best performing result, it's to perform a _controlled scientific experiment_ with a control group, from which you can learn things.
Occasionally there is pure scientific interest.
But far more frequently, the purpose of the A/B is to optimize the outcome.
This is why Google Analytics has exclusively chosen multi-armed bandit for its its A/B test framework.
- mobjack 7y agoIt doesn't have to be a pure science interest. If you want to use the results from one test to inform what to test next, then A/B tests are better optimizing for truth. Most website changes dont make a significant difference in conversion. If you use MAB, then you dont know if the winner is really better or the result of random variation.
- paulddraper 7y ago> If you use MAB, then you dont know if the winner is really better or the result of random variation. No, you know the outcome of MAB is the best. If it's clearly the best, it converges quickly. If it's less clearly the best, it converges slowly. Either way though, you haven't lost any more conversions that necessary to find that out, or needed to put in non-mathematically fail safes.