3 ms·

Evan, does this formula account for multiple comparisons (if you have multiple goals and multiple variations)? I guess it would suffer from the same problems th

Evan, does this formula account for multiple comparisons (if you have multiple goals and multiple variations)? I guess it would suffer from the same problems that if you have 100 variations and 10 goals, some of them are bound to produce a significant result, just by random chance. Isn't it? In classical testing, you can fix it by making your calculated alpha smaller than the real alpha, so you need much more data if there are multiple comparisons. What happens in Bayesian case? Edit: I did some Googling and found this http://www.stat.columbia.edu/~gelman/research/unpublished/multiple2.pdf http://www.stat.columbia.edu/~gelman/research/unpublished/mu...

grayclhn 12y ago
That paper is a very different setup --- the hierarchical model Gelman, Hill, and Yajima use implicitly builds multiple comparisons into their process. You could do the same thing here in principle by drawing alpha and beta for different experiments from an underlying distribution, but I'd be cautious about applying that approach mindlessly. If you haven't read it, Gelman and Hill's book is excellent: http://www.stat.columbia.edu/~gelman/arm/ http://www.stat.columbia.edu/~gelman/arm/ and so is Gelman's blog.