3 ms·

Thanks for doing this, Evan! Much appreciated - I've already implemented it to compare with our traditional confidence score based A/B metrics. I'm not sure wh

Thanks for doing this, Evan! Much appreciated - I've already implemented it to compare with our traditional confidence score based A/B metrics. I'm not sure what you mean by Monte Carlo for confidence bounds - is there some reading I can do to brush up on that? edit: here is your solution implemented in Ruby as I implemented it https://gist.github.com/nickelser/6dfa0f2737c502dae267 https://gist.github.com/nickelser/6dfa0f2737c502dae267

paraschopra 12y ago
Confidence interval means that is expected conversion rate of A is 0.5 and for B is 0.4, what is the actual spread. In other words, say, for 95% of the time does the conversion rate fall in the 0.3-0.5 range or 0.35-0.45 range. More data you collect , narrower this range will be. Confidence bounds for individual conversion rates can be calculated from the formula since they follow beta distribution. What Evan is saying that the confidence bound for the difference in conversion rates would need to be calculated through repeated simulations and taking (say) 5 percentile and 95 percentile of the values that come. For example, given you know probabilities for A and B (say 0.5 and 0.4) and that number of trials has been 100, you can simply run thousands of simulations, know what the conversion rates come out to be in respective simulations and aggregate the difference in conversion rates at the end of each simulation. That would be a confidence bound.
- octernion 12y ago
  Gotcha - thanks so much for writing this up. I'll see about actually implementing this :)