3 ms·
Agree, that is absolutely not how AB tests are run. No one runs tests sequentially on 100% of traffic - that would be like a McDonalds offering two versions of
by tqi 6y ago
Agree, that is absolutely not how AB tests are run. No one runs tests sequentially on 100% of traffic - that would be like a McDonalds offering two versions of the Egg McMuffin, one from 7a-11a and the second from 11a-6p, and declaring the v1 a clear winner because it had more sales.
Assignments are also almost certainly sticky based on a browser cookie.
- megablast 6y agoNo. It’s like if McDonald’s offered a in the first half hour, and b in the second half hour. I’m sure you would agree that would be a good representation.
- hntrader 6y agoThe traffic to a restaurant isn't uniformly distributed over an hour window. There's a peak 30-min in the morning just before work starts for most people, for example.
- tqi 6y agoSequential testing will never be truly random, unless you can guarantee that your traffic is uniform across time. Imagine you're running an AB test on a website, where the number of customers you get increases linearly over time. If you test two features sequentially, there is no duration you could pick that is small enough to where you would not expect the second option to win.
- deleted 6y ago[deleted]