5 ms·
What exactly makes you convinced that it works? To be specific: why wouldn’t there be bias in the A/B testing results, too? There are literally people who giv
by omginternets 4y ago
What exactly makes you convinced that it works? To be specific: why wouldn’t there be bias in the A/B testing results, too?
There are literally people who give astrological analyses for a living.
- mcswell 4y agoGreat minds! I was writing more or less the same thing, you beat me to publication by three minutes.
- lIl-IIIl 4y agoWe are talking about publication bias, where the decision whether to publish something is biased by the outcome of the experiment. I think this doesn't really apply to A/B testing, because people are incentivized pay as much attention to negative results as to positive ones.
- zeroonetwothree 4y agoFrom what I’ve seen there is even more incentive to focus on positive A/B tests. It’s the way you get credit for your work at a company. A negative test is counted as barely anything. So your incentive is to run tons of tests, then cherry pick only the positive ones and announce them widely. Another strategy is to track multiple metrics for each test and not adjust for that when computing p values. But then at the end you only report the one metric that was positive.
- aaaaaaaaaaab 4y agoPeople are incentivized to pay attention to the result that increases their mid-year bonus the most.
- zeroonetwothree 4y agoA/B testing has a ton of issues as well that make it easy to be fooled https://biggestfish.substack.com/p/data-as-placebo https://biggestfish.substack.com/p/data-as-placebo
- akyu 4y agoOf course.
- akyu 4y agoI cannot share the reason I am convinced it works. But I can tell you I am convinced it works. I'm sure many people here are in similar situations.