4 ms·

A/B testing significance calculator (spreadsheet in Excel)

19 points by joshuacc 16y ago

ashishk 16y ago
If you're interested in a web-based calculator, I've found this one to be useful: http://www.usereffect.com/split-test-calculator http://www.usereffect.com/split-test-calculator
- paraschopra 16y ago
  Yup, but do note that the calculator you linked is based on Chi-square and the one in excel is based on z-test. So, results may differ slightly! In case you need to use a web-based calculator (z-test), here is one on my site: http://visualwebsiteoptimizer.com/ab-split-significance-calculator/ http://visualwebsiteoptimizer.com/ab-split-significance-calc...
  - btilly 16y ago
    Why did you choose to use the z-test?
    - paraschopra 16y ago
      Z-test gives you confidence intervals on conversion rate with G-test or Chi-square test doesn't
      - btilly 16y ago
        Good point. Confidence intervals that shouldn't be trusted too much, but confidence intervals none the less.
- briancray 16y ago
  I really prefer Chapra's due to the cleanliness of the design and clarity of results.
nhebb 16y ago
It's been years since I last cracked a statistics book. Would a t-test be more accurate than a z-test or chi-squared for small samples? I've love to see a write-up on the statistics equations for A/B testing and guidelines on when to use which.
- btilly 16y ago
  Would a t-test be more accurate than a z-test or chi-squared for small samples? No. Here is a brief overview of the different tests. The z-test depends on the assumption that the sum of random events is well approximated by a normal distribution. This happens with a large enough sample size. But is wrong for small sample sizes. A further weakness with the z-test is that the same sample data is being used to estimate variance and averages. Depending on the specifics of your actual sample, both estimates can be off in a way that confuses the test. The t-test corrects for this possibility but assumes that the individual measurements you are making are normally distributed. It is quite sensitive to this assumption. Unfortunately in A/B testing the measurements are binary yes/no, 1/0 decisions, which are decidedly not normally distributed. The g-test is designed to be a very good approximation to the exact problem that A/B testing tries to solve. Unfortunately it requires you to compute logs, which was hard before computers. The chi-square test is designed as an easily computed good approximation to the g-test. It only needs you to square numbers and add them together. It therefore became the most widely used algorithm, and is the one that people have heard of. Now that we have computers, the g-test is the test we should use. But nobody has heard of it. (However you're still free to use it! And should.) If you get a large enough body of measurements then all tests converge to the z-test. I've love to see a write-up on the statistics equations for A/B testing and guidelines on when to use which. http://elem.com/~btilly/effective-ab-testing/ http://elem.com/~btilly/effective-ab-testing/ goes through this in some detail.
  - equark 16y ago
    A/B testing is a Bayesian decision problem. Hypothesis testing is really the wrong way to frame this problem. Dynamically it is a bandit problem and statically it's just about maximizing expected utility, where the expectation is taken with respect to the posterior distribution. This becomes especially crucial when doing lots of tests or experimenting sequentially. All the frequentist proposals don't have their advertised coverage rates in these situations. Unfortunately most write-ups seem to be completely ignorant of Bayesian statistics.

19 points by joshuacc 16y ago

If you're interested in a web-based calculator, I've found this one to be useful: http://www.usereffect.com/split-test-calculator http://www.usereffect.com/split-test-calculator

paraschopra 16y ago
Yup, but do note that the calculator you linked is based on Chi-square and the one in excel is based on z-test. So, results may differ slightly! In case you need to use a web-based calculator (z-test), here is one on my site: http://visualwebsiteoptimizer.com/ab-split-significance-calculator/ http://visualwebsiteoptimizer.com/ab-split-significance-calc...
- btilly 16y ago
  Why did you choose to use the z-test?
  - paraschopra 16y ago
    Z-test gives you confidence intervals on conversion rate with G-test or Chi-square test doesn't
    - btilly 16y ago
      Good point. Confidence intervals that shouldn't be trusted too much, but confidence intervals none the less.
briancray 16y ago
I really prefer Chapra's due to the cleanliness of the design and clarity of results.

nhebb 16y ago

It's been years since I last cracked a statistics book. Would a t-test be more accurate than a z-test or chi-squared for small samples? I've love to see a write-up on the statistics equations for A/B testing and guidelines on when to use which.

btilly 16y ago
Would a t-test be more accurate than a z-test or chi-squared for small samples? No. Here is a brief overview of the different tests. The z-test depends on the assumption that the sum of random events is well approximated by a normal distribution. This happens with a large enough sample size. But is wrong for small sample sizes. A further weakness with the z-test is that the same sample data is being used to estimate variance and averages. Depending on the specifics of your actual sample, both estimates can be off in a way that confuses the test. The t-test corrects for this possibility but assumes that the individual measurements you are making are normally distributed. It is quite sensitive to this assumption. Unfortunately in A/B testing the measurements are binary yes/no, 1/0 decisions, which are decidedly not normally distributed. The g-test is designed to be a very good approximation to the exact problem that A/B testing tries to solve. Unfortunately it requires you to compute logs, which was hard before computers. The chi-square test is designed as an easily computed good approximation to the g-test. It only needs you to square numbers and add them together. It therefore became the most widely used algorithm, and is the one that people have heard of. Now that we have computers, the g-test is the test we should use. But nobody has heard of it. (However you're still free to use it! And should.) If you get a large enough body of measurements then all tests converge to the z-test. I've love to see a write-up on the statistics equations for A/B testing and guidelines on when to use which. http://elem.com/~btilly/effective-ab-testing/ http://elem.com/~btilly/effective-ab-testing/ goes through this in some detail.
- equark 16y ago
  A/B testing is a Bayesian decision problem. Hypothesis testing is really the wrong way to frame this problem. Dynamically it is a bandit problem and statically it's just about maximizing expected utility, where the expectation is taken with respect to the posterior distribution. This becomes especially crucial when doing lots of tests or experimenting sequentially. All the frequentist proposals don't have their advertised coverage rates in these situations. Unfortunately most write-ups seem to be completely ignorant of Bayesian statistics.