3 ms·
Thank you for this. I especially appreciated that you have brought the concept of "publication bias" (though not by name) into the discussion of AB testing. In
by carbocation 16y ago
Thank you for this. I especially appreciated that you have brought the concept of "publication bias" (though not by name) into the discussion of AB testing. In medicine there is a tendency to publish positive results and not negative results, leading to a bias in favor of positive results. Most likely, this is an equally large or larger problem for bloggers.
- gsaines 16y agoHey Carbocation, yeah, the file-drawer effect can be particularly strong with results for A/B and multivariate testing and partially I wanted to make sure people knew it was hard to get right and also to spread the word about our folly in the hopes it might save others time! :)
- patio11 16y agoYep. I have an A/B results page which automatically collects closed tests. http://abingo.org/results http://abingo.org/results 75% or so of tests get a null result (A may or may not be better than B.). Lots of the remainder tell me to what degree of statistical significance my great new idea for improvement sucked. But you can still improve over time.
- gsaines 16y agoHey Patio, thanks for sharing that link, I didn't even know that page existed!
- carbocation 16y agoOh, that is awesome! Do you adjust for multiple testing? You could do so and then test an overarching hypothesis (say, that there is at least one test on that page that differs from what would be expected by chance when looking at N tests).