7 ms·
Multiple control groups will highlight the degree that a result doesn't have statistical significance yet. We know that peeking is a problem[1], e.g. you peek a
by infogulch 11y ago
Multiple control groups will highlight the degree that a result doesn't have statistical significance yet. We know that peeking is a problem[1], e.g. you peek and see that B is outperforming A by 4% you might be more inclined to stop than if you saw that A is also outperforming A by 4%; it makes you stop and think.
Re: N buckets: you'll get diminishing returns and it will take N times more data points to get a statistically significant result in any bucket.
[1]: http://www.evanmiller.org/how-not-to-run-an-ab-test.html http://www.evanmiller.org/how-not-to-run-an-ab-test.html
- btilly 11y agoSo you have a random chance of noticing what statistics ALREADY tells you. Except that only sometimes do you notice the problem by chance, and statistics can let you always know about it. This is a net loss.
- infogulch 11y agoOh I totally agree, people should just be using statistics correctly. But I would argue that the poor tool that's used is more effective than the perfect tool that's never touched. Perhaps it would be a teaching tool/stepping stone to get people to understand and use statistics correctly.