3 ms·

These are great! Thank you Evan. Your sample size calculator is wonderful and beats the hell out of the 90s tool I've been using which cautions me that my brows

These are great! Thank you Evan. Your sample size calculator is wonderful and beats the hell out of the 90s tool I've been using which cautions me that my browser must support "JavaScript" to use it :) As an alternative to the Chi-squared calculator, people might want to check out ABBA, a tool I wrote here at Thumbtack: http://www.thumbtack.com/labs/abba/ http://www.thumbtack.com/labs/abba/ It shares the visual component and the linkability, two great features you've nailed. It lacks the live updating and the slider, which is really cool and something I've wanted to add to ABBA for a long time. On the other hand, it supports multiple groups compared against the baseline simultaneously and incorporates a correction for multiple testing into its p-values and confidence intervals, which can be handy. It also uses different mathematics under the hood, but that's not going to be a concern for most users. Glad to see another step towards a more statistically-aware world!

gostevehoward 13y ago
Small follow-up question: I'm curious how your sample size calculator chooses its value. Traditional power analysis asks for two proportions rather than for a proportion and an effect size. So the sample size for a positive effect differs from that for a negative effect (from the same baseline). I'd imagine your tool would conservatively present the larger of the two. However, it seems to present something near the midpoint of the two. Is that the intention? Or is there some other statistics being used here? (For example, with an 8% baseline conversion rate, a 1% absolute detectable different, 85% power and 10% significance level, your tool says 10,583 per branch. R's `power.prop.test` gives sample sizes of 11,182 for a positive change (8% vs 9%) and 9,974 for a negative change (8% vs 7%). The exact midpoint is 10,578.)
yahelc 13y ago
You'll be interested to hear that the digital analytics team at the Obama campaign sometimes made use of your tool for sharable A/B calculations. I even built a bookmarklet for quickly grabbing numbers off off of a page (usually for Google Analytics data) and passing them to your calculator. https://github.com/yahelc/ABBA-bookmarklet https://github.com/yahelc/ABBA-bookmarklet
- gostevehoward 13y ago
  Wow this bookmarklet is awesome! I'm going to start using it. Thanks!

gostevehoward 13y ago

Small follow-up question: I'm curious how your sample size calculator chooses its value. Traditional power analysis asks for two proportions rather than for a proportion and an effect size. So the sample size for a positive effect differs from that for a negative effect (from the same baseline). I'd imagine your tool would conservatively present the larger of the two. However, it seems to present something near the midpoint of the two. Is that the intention? Or is there some other statistics being used here? (For example, with an 8% baseline conversion rate, a 1% absolute detectable different, 85% power and 10% significance level, your tool says 10,583 per branch. R's `power.prop.test` gives sample sizes of 11,182 for a positive change (8% vs 9%) and 9,974 for a negative change (8% vs 7%). The exact midpoint is 10,578.)

yahelc 13y ago

You'll be interested to hear that the digital analytics team at the Obama campaign sometimes made use of your tool for sharable A/B calculations. I even built a bookmarklet for quickly grabbing numbers off off of a page (usually for Google Analytics data) and passing them to your calculator. https://github.com/yahelc/ABBA-bookmarklet https://github.com/yahelc/ABBA-bookmarklet

gostevehoward 13y ago
Wow this bookmarklet is awesome! I'm going to start using it. Thanks!