3 ms·

I'm interested to see the sample size calculations. Those numbers don't jibe with any calculations I've ever done.

• throwawayg99 13y ago
  Nor me, to collect a 1% change: > power.prop.test(p1=0.1, p2=0.11, power=0.8,sig.level=0.05) Two-sample comparison of proportions power calculation n = 14750.79 p1 = 0.1 p2 = 0.11 sig.level = 0.05 power = 0.8 alternative = two.sided NOTE: n is number in *each* group Edit: so that's 30,000 samples. To do the same with a 5% change in the same area is still 1300 samples (alpha=0.8, p=0.05)
  • noelwelsh 13y ago
    I normally consider relative minimum discernable effect. Your 5% absolute change is a 50% increase is the base rate. I also typically go for a higher power (e.g. 0.9). Under these conditions 60K samples is more typical. Sample size calculator here: http://www.evanmiller.org/ab-testing/sample-size.html http://www.evanmiller.org/ab-testing/sample-size.html You'll need to change the defaults to match the above to get the figures I mention.
    • throwawayg99 13y ago
      You're right. That's a bit more reasonable, so back to my 10% change in base rate (1% abs.) but with a 90% power: > power.prop.test(p1=0.1, p2=0.11, power=0.9,sig.level=0.05) Two-sample comparison of proportions power calculation n = 19746.62 p1 = 0.1 p2 = 0.11 sig.level = 0.05 power = 0.9 alternative = two.sided NOTE: n is number in *each* group Requires about 40,000 samples per test. I would strongly recommend anyone serious about doing this look in to MAB testing, as A-B testing is way too expensive for reasonable scale testing (unless you have a strong a priori hypothesis to test).
  • deleted 13y ago
    [deleted]