6 ms·

Perhaps it was just that the correct, holistic metrics were not used to judge the a/b test?

• btilly 14y ago
  It could be that the A/B test results were improperly stopped too early to be meaningful. (Hint: stopping when you hit 95% is not enough.) It could be, as they think, that the group that they were testing is not the group that they needed to target for future growth. It could be that there are familiarity effects that threw off the test. It could be that they were using the wrong metrics for what they needed. It could be that to get at what they want, they'd need to carefully segment their test in ways that they didn't know how to do. There are a lot of ways to not get the answers you were looking for out of your A/B tests. Of course it is also possible that they don't realize that their growth would have been better had they paid attention to the A/B test. We have a tendency towards confirmation bias - whatever decision we make, we often will look for ways to convince ourselves that the decision we made is correct. We're not given enough information to distinguish between these possibilities.