4 ms·
It's sensitive to your volume, conversion rates, and the magnitude of the difference in conversion rates between A and B. In general, volume is good, higher co
by patio11 15y ago
It's sensitive to your volume, conversion rates, and the magnitude of the difference in conversion rates between A and B. In general, volume is good, higher conversion rates (or easier intermediate conversions, like to email submit rather than to purchase) are good, and large differences in conversion rates are good. It would take a lot of time to statistically significantly discriminate between .001% and .0012% on volumes of 5 visitors a day.
People like easy answers, though, so my easy answer is "A/B testing is worthwhile if you have 100 visitors a day. Otherwise, spend the bulk of your time finding 100 visitors a day first."
- arnorhs 15y agoIs 100 a day enough? Wouldn't it take forever to get statistical significance? (unless the results are really dramatic)
- metachris 15y agoPerhaps. Let's say you run the test for 5 days (500 users), and 80% of conversions happen on variant A, it does provide you with something to work with and experiment further.
- LargeWu 15y agoIt depends on how much variance you normally have, and the size of the change you hope to measure. http://en.wikipedia.org/wiki/Statistical_power http://en.wikipedia.org/wiki/Statistical_power