3 ms·
> Some experimenters at Twitter proposed running A/A/B instead of A/B tests, to reduce the false positive rate due to “biased” buckets. An A/A/B test is one in
by gwern 11y ago
> Some experimenters at Twitter proposed running A/A/B instead of A/B tests, to reduce the false positive rate due to “biased” buckets. An A/A/B test is one in which in addition to the provided control (A1), an experimenter would add another control group (A2) to their experiment.
I wonder what intuitions drove that suggestion? Would they be even happier with 3 buckets (each a third of the size)? With 4 (quarters)? 5 (fifths)? ..._n_ buckets, 1 for each sample? At what point did those experimenters think it would stop being an improvement over a single control group?
- infogulch 11y agoMultiple control groups will highlight the degree that a result doesn't have statistical significance yet. We know that peeking is a problem[1], e.g. you peek and see that B is outperforming A by 4% you might be more inclined to stop than if you saw that A is also outperforming A by 4%; it makes you stop and think. Re: N buckets: you'll get diminishing returns and it will take N times more data points to get a statistically significant result in any bucket. [1]: http://www.evanmiller.org/how-not-to-run-an-ab-test.html http://www.evanmiller.org/how-not-to-run-an-ab-test.html
- btilly 11y agoSo you have a random chance of noticing what statistics ALREADY tells you. Except that only sometimes do you notice the problem by chance, and statistics can let you always know about it. This is a net loss.
- infogulch 11y agoOh I totally agree, people should just be using statistics correctly. But I would argue that the poor tool that's used is more effective than the perfect tool that's never touched. Perhaps it would be a teaching tool/stepping stone to get people to understand and use statistics correctly.
- jfarmer 11y agoSpeaking from experience, the people suggesting such things aren't thinking that way.
- btilly 11y agoThe intuition undoubtedly is, "I don't understand the fancy math, let's do something I think I understand instead."