If you're running an A/B test, by which I mean precisely that you already have two versions available and are simply trying to decide which is most likely to gi

If you're running an A/B test, by which I mean precisely that you already have two versions available and are simply trying to decide which is most likely to give better results in the future by comparing how successfully they achieve the desired result with all else being equal, then I'm still waiting for someone to explain to me why the correct question is not simply "Which performs better, A or B?" The statistical rigour of all the hypothesis tests and Bayesian methods and so on is laudable, but fundamentally any result and any conclusion you draw from those tests can only ever be as good as the underlying assumptions from which the result was derived. For example, if you choose to perform any statistical test that designates one of A and B the default case and tests for evidence in favour of the other, you have made a determination that the situation is not symmetric. You can choose a level of power or a significance level you want to achieve, but the 5% convention is again arbitrary. The same goes for any prior you choose in a Bayesian method. Fundamentally, if you are truly starting from a neutral situation where you just want to know which of A and B is better, and you truly have no other information to go on other than how they have performed since the start of your measurements with all else being equal, then it remains the case that you still have nothing else to go on no matter what statistical calculations you perform.