3 ms·
I have not worked in any company doing ab tests like that, it is always base vs variant. Option 1 vs option 2/3 and then if the feature wins, you can try option
by throwaway47292 5y ago
I have not worked in any company doing ab tests like that, it is always base vs variant. Option 1 vs option 2/3 and then if the feature wins, you can try option 2 vs option 3, if not they would scrap it, or blame it on the implementation and keep retrying until they either get false positive, or survivor bias makes it positive.
I am not defending it as I also believe that ab tests ruin the web, but if anyone does option2/3 at start they are not doing AB test at all.
- Karrot_Kream 5y agoI agree. I've seen a lot of AB test abuse, but it's mostly not outright, the kind where you try to re-roll until you get a false positive or sample sizes are so small that you get an effect on the treatment group despite the control group being much larger. It's still all statistical garbage though.