3 ms·
Note: I did not A/B test the experiment covered in the Mixpanel article. I did just release an A/B testing framework (A/Bingo) today. You can find it here: h
by patio11 17y ago
Note: I did not A/B test the experiment covered in the Mixpanel article. I did just release an A/B testing framework (A/Bingo) today. You can find it here:
http://news.ycombinator.com/item?id=764508 http://news.ycombinator.com/item?id=764508
1) I run tests until I get statistically significant results. How long that takes is outside the scope of this comment, except to say that if you're testing something with a high conversion rate and testing two very different alternatives you can get significant results in ~250 users or so. This lets me do very rapid iterating. Changes in my core interaction can be A/B tested in under a week, which works well for me (I do most development on weekends).
2) I run lots of tests in parallel. For why that is not as bad an idea as you might think it is, see the A/Bingo FAQ.
3) That is a core design consideration for A/Bingo. I obsessively track users to make sure they always get the same alternative. A/Bingo is the only Rails A/B testing framework that does this, to my knowledge.
- jfarmer 17y agoIt's also easy to test things with very low conversion rates. The worst conversion rate to have in terms of required sample size is 50%.