> When doing a test you should plan it in advance based on other data I think this is not entirely true. In areas where experimentation feedback is fast (and e

> When doing a test you should plan it in advance based on other data I think this is not entirely true. In areas where experimentation feedback is fast (and experiments are probably cheaper to run), the problem is much more accurately and sample efficiently solved by Thompson Sampling [1,2] which in fact dictates you to have a broad enough prior over the solution space and then let the posterior dictate your conclusions. [1]: https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/thompson.pdf https://www.microsoft.com/en-us/research/wp-content/uploads/... [2]: http://www.economics.uci.edu/~ivan/asmb.874.pdf http://www.economics.uci.edu/~ivan/asmb.874.pdf