4 ms·

Thanks for this great article and thanks for sharing the code! Methinks I'll be installing it sometime in the next week. Quick questions about your A/B testing

by swombat 17y ago

Thanks for this great article and thanks for sharing the code! Methinks I'll be installing it sometime in the next week. Quick questions about your A/B testing... * What kind of user volumes are you looking to do A/B testing on? How does that relate to how long each of your tests runs/will run? * Are you running several A/B tests in parallel? * Do you make each user an A or B, or do you keep the A/B testing page-based? I'm assuming the former, since otherwise the same user might see both sides of the test, which is not desirable.

patio11 17y ago
Note: I did not A/B test the experiment covered in the Mixpanel article. I did just release an A/B testing framework (A/Bingo) today. You can find it here: http://news.ycombinator.com/item?id=764508 http://news.ycombinator.com/item?id=764508 1) I run tests until I get statistically significant results. How long that takes is outside the scope of this comment, except to say that if you're testing something with a high conversion rate and testing two very different alternatives you can get significant results in ~250 users or so. This lets me do very rapid iterating. Changes in my core interaction can be A/B tested in under a week, which works well for me (I do most development on weekends). 2) I run lots of tests in parallel. For why that is not as bad an idea as you might think it is, see the A/Bingo FAQ. 3) That is a core design consideration for A/Bingo. I obsessively track users to make sure they always get the same alternative. A/Bingo is the only Rails A/B testing framework that does this, to my knowledge.
- jfarmer 17y ago
  It's also easy to test things with very low conversion rates. The worst conversion rate to have in terms of required sample size is 50%.