2 ms·
If you run out of experiment space, the solution is to do overlapping experiments, not to avoid running experiments at all. This biases your data somewhat, so
by nostrademons 17y ago
If you run out of experiment space, the solution is to do overlapping experiments, not to avoid running experiments at all. This biases your data somewhat, so it's not ideal. But by comparison - nearly every experiment at Amazon is 50/50: they try it one way on half of users, and another way on the other half. And they overlap, so you're running lots of experiments at once. Not absolutely great for data collection, but it doesn't seem to be much of a problem in practice for them.