3 ms·

I agree with other critiques here that this A/B testing calculator does little to add to the conversation, and someone who uses it would be mislead in how to in

by martian 13y ago

I agree with other critiques here that this A/B testing calculator does little to add to the conversation, and someone who uses it would be mislead in how to interpret results. The procedure my team uses is: 1) hypothesize an expected conversion rate and whether to use a one-sided test (if we're testing that the state of the world remains unchanged) or a two-sided test (if we're testing that that the state of the world has changed as a result of the variation) 2) run those numbers through this power/sample size calculator to determine the number of visitors we need before we can analyze the experiment: http://www.stat.ubc.ca/~rollin/stats/ssize/b2.html http://www.stat.ubc.ca/~rollin/stats/ssize/b2.html 3) wait for traffic 4) after enough visitors have come to the funnel, pass the resulting conversion numbers through ABBA http://www.thumbtack.com/labs/abba/ http://www.thumbtack.com/labs/abba/ [1] to see confidence intervals on our results For further reading, I highly recommend: http://visualwebsiteoptimizer.com/split-testing-blog/how-to-calculate-ab-test-sample-size/ http://visualwebsiteoptimizer.com/split-testing-blog/how-to-... [1] disclaimer: my colleague wrote ABBA

jessicaraygun 13y ago
Hey martian, thanks for the feedback. Your methodology is pretty sound - it looks like we use a similar technique. If you supply a small sample size here, you'll get the message, "There is not enough data to determine a conclusive result".
blueblob 13y ago
I also agree. This is effectively 2 lines of R code: binom.test(500,2100) binom.test(1000,2000) With a boxplots/confidence interval plots (which are not many more lines). Whether this is useful or not depends on how much people know about statistics I guess.
- jrpt 13y ago
  This calculator makes reasonable assumptions. For example, it doesn't use bootstrapping, and it assumes a normal distribution. I don't see how making these assumptions makes the calculator bad, it just means if you are a stats person who wants to do something different, you'll need to implement your own script. Outside of the HN crowd, there are people who don't code, who nonetheless work with a/b tests, and still need a measure of statistical confidence. Since a binomial distribution approaches a normal distribution for large N, that's a valid assumption to make. The arguments in this thread over which is the precise distribution are over analyzing, in my opinion. If I had an experiment that gave one winner with a binomial distribution, and a different winner with the normal approximation, I'd feel little confidence in the results and want to run it again or collect more data. I don't see how that would happen though. Bootstrapping would be better, sure, but it would be confusing for an a/b test calculator to give you different numbers each time you refresh the page. You're free to write your own R scripts if you want to do it yourself. I agree with martian's advice about figuring out a minimum number of visitors you expect to need before analyzing the data, or else you're at risk of biasing the experiment by declaring a winner prematurely.

by martian 13y ago

jessicaraygun 13y ago

Hey martian, thanks for the feedback. Your methodology is pretty sound - it looks like we use a similar technique. If you supply a small sample size here, you'll get the message, "There is not enough data to determine a conclusive result".

blueblob 13y ago

I also agree. This is effectively 2 lines of R code: binom.test(500,2100) binom.test(1000,2000) With a boxplots/confidence interval plots (which are not many more lines). Whether this is useful or not depends on how much people know about statistics I guess.

jrpt 13y ago
This calculator makes reasonable assumptions. For example, it doesn't use bootstrapping, and it assumes a normal distribution. I don't see how making these assumptions makes the calculator bad, it just means if you are a stats person who wants to do something different, you'll need to implement your own script. Outside of the HN crowd, there are people who don't code, who nonetheless work with a/b tests, and still need a measure of statistical confidence. Since a binomial distribution approaches a normal distribution for large N, that's a valid assumption to make. The arguments in this thread over which is the precise distribution are over analyzing, in my opinion. If I had an experiment that gave one winner with a binomial distribution, and a different winner with the normal approximation, I'd feel little confidence in the results and want to run it again or collect more data. I don't see how that would happen though. Bootstrapping would be better, sure, but it would be confusing for an a/b test calculator to give you different numbers each time you refresh the page. You're free to write your own R scripts if you want to do it yourself. I agree with martian's advice about figuring out a minimum number of visitors you expect to need before analyzing the data, or else you're at risk of biasing the experiment by declaring a winner prematurely.