This calculator makes reasonable assumptions. For example, it doesn't use bootstrapping, and it assumes a normal distribution. I don't see how making these assu

This calculator makes reasonable assumptions. For example, it doesn't use bootstrapping, and it assumes a normal distribution. I don't see how making these assumptions makes the calculator bad, it just means if you are a stats person who wants to do something different, you'll need to implement your own script. Outside of the HN crowd, there are people who don't code, who nonetheless work with a/b tests, and still need a measure of statistical confidence. Since a binomial distribution approaches a normal distribution for large N, that's a valid assumption to make. The arguments in this thread over which is the precise distribution are over analyzing, in my opinion. If I had an experiment that gave one winner with a binomial distribution, and a different winner with the normal approximation, I'd feel little confidence in the results and want to run it again or collect more data. I don't see how that would happen though. Bootstrapping would be better, sure, but it would be confusing for an a/b test calculator to give you different numbers each time you refresh the page. You're free to write your own R scripts if you want to do it yourself. I agree with martian's advice about figuring out a minimum number of visitors you expect to need before analyzing the data, or else you're at risk of biasing the experiment by declaring a winner prematurely.