4 ms·
For an A/A/B test are you taking three samples (instead of two), and two of the three get shown the same thing (A)? Then you only consider the results for the B
by pottertheotter 5y ago
For an A/A/B test are you taking three samples (instead of two), and two of the three get shown the same thing (A)? Then you only consider the results for the B group if the two A groups show the same behavior?
- ComputerGuru 5y agoNot the person you're responding to, but yes, that's the idea. It's a control not for the the B but for the unknown unknowns that may or may not be there. If A' and B both statistically differ from A, then you have a problem because you're not testing what you think you are testing, regardless of what your naive A/B test's p-value would have indicated.
- Maxion 5y agoYou take three samples and two of three get shown the same thing. What happens here is both A groups will show different results until your sample grows enough that the CI window becomes small. This helps to show the effect of a low sample from a non-uniform distribution. A lot of people (me included) think they know statistics, but they don't. The blogpost in OP also tries to explain the same thing - you shouldn't do statistics without understanding what it is your doing.
- bananamansion 5y agodont you then need to run the experiment for a long long time to get the significance of it? + the site needs enough users viewing the page to run it in any reasonable amount of time.. i would think most sites dont get enough traffic to run a/a let alone ab?