3 ms·

This is part of the reasons why A/B testing might fail in reality, it is not just about collecting random number of responses about A and B. When doing a test

• mumblemumble 6y ago
  A concrete example: For starters, consider this xkcd comic: https://xkcd.com/882/ https://xkcd.com/882/ If you're doing A/B testing where you continuously watch the results, and conclude your test as soon as you see the numbers significantly deviate, what you're doing is more-or-less equivalent to what's happening in the comic, only it's the number of jelly beans that you vary instead of their color. Not exactly the same, since in the numbers case your draws from the random variable aren't independent, but still.
• activatedgeek 6y ago
  > When doing a test you should plan it in advance based on other data I think this is not entirely true. In areas where experimentation feedback is fast (and experiments are probably cheaper to run), the problem is much more accurately and sample efficiently solved by Thompson Sampling [1,2] which in fact dictates you to have a broad enough prior over the solution space and then let the posterior dictate your conclusions. [1]: https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/thompson.pdf https://www.microsoft.com/en-us/research/wp-content/uploads/... [2]: http://www.economics.uci.edu/~ivan/asmb.874.pdf http://www.economics.uci.edu/~ivan/asmb.874.pdf