3 ms·
In many cases, I think this is fine. Take the case given in the article: 2 designs that appear equal, but segmenting reveals (incorrectly) that one performs be
by trjordan 12y ago
In many cases, I think this is fine.
Take the case given in the article: 2 designs that appear equal, but segmenting reveals (incorrectly) that one performs better on Android and the other on iPhone. Statistically, it doesn't matter which one you pick, because you don't have a strong result. Anecdotally, you might go down the wrong path and waste some time doing something like showing design A to Android and B to iPhone. But the only thing lost is time -- probably a small amount of time compared to creating the different designs.
Best case, you realize there's something technically wrong with your changes in a certain segment because things are widely different. Worst case, you make some changes that don't matter. Overall, the risk / reward of segmenting seems worthwhile.
- yummyfajitas 12y agoThis is very true - my post is mainly coming from a classical frequentist hypothesis testing perspective. From a Bayesian/portfolio optimization perspective, all that segmentation isn't hurting you much. But from a business perspective, it's a huge distraction. Your marketers shouldn't be wasting time segmenting - there is no reason to believe it's making any money. And due to multiple comparisons, it's virtually guaranteed that your A/B tests will steer the marketers in the direction of more segmentation.