You would be better served by a Bayesian approach to A/B testing, and measure directly the probability of A converting more than B. http://www.evanmiller.org/ba

You would be better served by a Bayesian approach to A/B testing, and measure directly the probability of A converting more than B. http://www.evanmiller.org/bayesian-ab-testing.html http://www.evanmiller.org/bayesian-ab-testing.html You can then apply some sort of decision rule such as the difference being above a certain threshold.