2 ms·
> Varian's point is that you get diminishing returns after a certain point with more data. That's true only if you have a fixed number of questions. He points
by anamax 17y ago
> Varian's point is that you get diminishing returns after a certain point with more data.
That's true only if you have a fixed number of questions.
He points out that Google can do an experiment with 0.5% of its daily searches that would take Microhoo 2% of their daily searches and concludes that Google has no advantage.
That tells us that Google can do 4x as many experiments as MicroHoo. Does anyone really believe that Google has enough search traffic to do every worthwhile experiment?
- nostrademons 17y agoIf you run out of experiment space, the solution is to do overlapping experiments, not to avoid running experiments at all. This biases your data somewhat, so it's not ideal. But by comparison - nearly every experiment at Amazon is 50/50: they try it one way on half of users, and another way on the other half. And they overlap, so you're running lots of experiments at once. Not absolutely great for data collection, but it doesn't seem to be much of a problem in practice for them.