3 ms·
" If you have a long running test, just to gather enough traffic, changes are some other factors have changed as well, like the weather, weekdays / weekends, ti
by teekert 11y ago
" If you have a long running test, just to gather enough traffic, changes are some other factors have changed as well, like the weather, weekdays / weekends, time of month, etc."
I'd serve them at the same time, randomize which clients see which one.
- chii 11y agoAnd it turns out you randomly assigned B to a poorer user more often than not, and this make B worse...
- eru 11y agoThat's why you do it properly randomized. Statistics tell you how much of a risk of your scenario (and similar) you are running at most.
- spiderfarmer 11y agoOfcourse, that's what A/B testing is about. But this only works if you have lots of traffic. The smaller the amount of traffic, the bigger the influence of offsite factors. Also, the results become more difficult to explain: https://news.ycombinator.com/item?id=11438098 https://news.ycombinator.com/item?id=11438098