4 ms·

Pretty animation, but I have some reservations about the maths: - Normal approximation, as already noticed, ain't no good. Use the Wilson score instead. - No

Pretty animation, but I have some reservations about the maths: - Normal approximation, as already noticed, ain't no good. Use the Wilson score instead. - No power calculation? Type II errors are far more important IMHO in typical web applications because switching costs are small. - non-overlapping 95% confidence interval do not imply p < 0.05. It's actually much lower than that. 83% CI is more like p of 0.05. (Errors add in quadrature.) There is a tension between making something simple for the lay person and providing knobs for the expert to twiddle. I can see the case for removing the knobs but the choices should at least be documented. [It's late here so this post is a bit slim on details. If you're interested sign up to http://bandits.mynaweb.com/ http://bandits.mynaweb.com/ as the next section covers confidence intervals.]

jessicaraygun 13y ago
Hey Noel, I was in charge of the math here - I really appreciate your feedback. You seem really knowledgeable in this area and I would love to improve this A/B testing library. If you have some time to chat, email me at jessica@sendwithus.com
keithwinstein 13y ago
Arguably the "best" confidence interval for this situation is the Blyth-Still-Casella interval (preferred by StatXact), and the "best" hypothesis test is Barnard's test. Here is code to calculate both: https://github.com/keithw/biostat https://github.com/keithw/biostat I say "arguably" literally -- there is a huge body of literature on confidence intervals for binary proportions, much of it in disagreement about what is important. The Wilson score interval and Agresti-Caffo and whatever else are fine approximate methods that came of age when ease of calculation was a big concern. But if you have a computer and you're baking one thing into a library, may as well make it the best one you can. Of course there is also plenty of merit to just picking some prior distribution and integrating over the conditional probability distribution given the data, aka a Bayesian approach. In practice I don't think this (stats geekery about the merits of different confidence or credible intervals) is the most important part. The numerical results from all these techniques will be pretty similar. The important part is in the design of the experiment, the interpretation, and playing by the rules. If you want to dynamically tune a Web site to make the most money as more information rolls in, that calls for a different experiment than a standard hypothesis test. (Even if you want to peek early at the results and possibly abort the test as a result, that calls for different tools and different rules.)
- noelwelsh 13y ago
  Hadn't heard of the Blyth-Still-Casella interval or Barnard's test before, so thanks for that. By "If you want to dynamically tune a Web site to make the most money as more information rolls in, that calls for a different experiment than a standard hypothesis test." you're talking about minimising regret / bandit algorithms?

jessicaraygun 13y ago

Hey Noel, I was in charge of the math here - I really appreciate your feedback. You seem really knowledgeable in this area and I would love to improve this A/B testing library. If you have some time to chat, email me at jessica@sendwithus.com

keithwinstein 13y ago

Arguably the "best" confidence interval for this situation is the Blyth-Still-Casella interval (preferred by StatXact), and the "best" hypothesis test is Barnard's test. Here is code to calculate both: https://github.com/keithw/biostat https://github.com/keithw/biostat I say "arguably" literally -- there is a huge body of literature on confidence intervals for binary proportions, much of it in disagreement about what is important. The Wilson score interval and Agresti-Caffo and whatever else are fine approximate methods that came of age when ease of calculation was a big concern. But if you have a computer and you're baking one thing into a library, may as well make it the best one you can. Of course there is also plenty of merit to just picking some prior distribution and integrating over the conditional probability distribution given the data, aka a Bayesian approach. In practice I don't think this (stats geekery about the merits of different confidence or credible intervals) is the most important part. The numerical results from all these techniques will be pretty similar. The important part is in the design of the experiment, the interpretation, and playing by the rules. If you want to dynamically tune a Web site to make the most money as more information rolls in, that calls for a different experiment than a standard hypothesis test. (Even if you want to peek early at the results and possibly abort the test as a result, that calls for different tools and different rules.)

noelwelsh 13y ago
Hadn't heard of the Blyth-Still-Casella interval or Barnard's test before, so thanks for that. By "If you want to dynamically tune a Web site to make the most money as more information rolls in, that calls for a different experiment than a standard hypothesis test." you're talking about minimising regret / bandit algorithms?