3 ms·
I'm reminded of the time I saw some A/B test results that didn't make much sense, but were highly significant[1] I asked how many A/B tests they were running..
by jaggederest 2y ago
I'm reminded of the time I saw some A/B test results that didn't make much sense, but were highly significant[1]
I asked how many A/B tests they were running... hundreds. Overlapping. At least they had a holdout group (that they mostly ignored, which indicated that all the A/B tests more or less made no difference)
[1] P < 0.001 with a large effect size. No, your a/b test probably didn't break the laws of economics - you probably messed up your data.
- MostlyStable 2y agoIf they were running many concurrent, overlapping A/B tests, then they didn't necessarily mess up their data. You are likely to get that result, honestly and truthfully, purely by chance, if you run enough tests. Unless the "running concurrent tests and not correcting your significance level", is what you meant by messing up their data, in which case yeah.
- jaggederest 2y agoYes precisely, the latter. I know there are techniques to disentangle that kind of thing but neither I nor they had them readily to hand.