3 ms·
Optimizely being for large enterprises, curious how people do A/B tests at their respective startups. Do most roll their own? How do you make sure your science
by raphaelrk 8y ago
Optimizely being for large enterprises, curious how people do A/B tests at their respective startups. Do most roll their own? How do you make sure your science is sound?
- t3scrote 8y agoWe set audience criteria where the user account must be created after the test launches, from there its a 50/50 split control/treatment experience (based on user id). The metric we are optimizing for is almost always conversion rate. We will turn the experiment off early if the treatment group is having really poor numbers, otherwise once about 4000 accounts have been entered into the experiment we plug the numbers into a bayesian calculator, and call it a winner of there is a 90%+ probability that the treatment beats the control. https://www.abtestguide.com/bayesian/ https://www.abtestguide.com/bayesian/
- tzahola 8y agoSo one in ten of your findings is bogus.
- t3scrote 8y agoBut isn’t that better than blindly introducing changes without testing them at all?
- joshuamorton 8y agoWhy so high? Beysian results aren't p-values, a 60-70% probably that treatment beats control is just that, not a pvalue of .4 or .3 (which would say nothing).