While it may be possible take the frequentist approach to AB testing, Bayesian inference is becoming the way to go with this.[1] Instead of directly setting up

While it may be possible take the frequentist approach to AB testing, Bayesian inference is becoming the way to go with this.[1] Instead of directly setting up a yes-no hypothesis test with the nearly impossible to use correctly p-values[2], Bayesian approaches aim to directly estimate whatever quantity you want. With the Bayesian approach, you get estimates for A, B, and A-B (or whatever combination you want, e.g. (A-B)/A). Each of those estimates are properly called posterior probability distributions and describe the range of possible values. The end result is that instead of saying "A is better than B (p < 0.05)" you get a probability distribution of A-B. From that probability distribution you can answer any question you want: the most likely difference between A and B (the average), the probability that A is better than B (just integrate the area above 0), or whatever is needed to make a decision. [1] https://conversionxl.com/blog/bayesian-frequentist-ab-testing/ https://conversionxl.com/blog/bayesian-frequentist-ab-testin... [2] https://www.amstat.org/asa/files/pdfs/p-valuestatement.pdf https://www.amstat.org/asa/files/pdfs/p-valuestatement.pdf