3 ms·
Are you sure? Consider the case where iOS users are more likely to convert than Android users, but you currently have very few iOS users. You then A/B test a n
by BoppreH 3y ago
Are you sure?
Consider the case where iOS users are more likely to convert than Android users, but you currently have very few iOS users. You then A/B test a new design that imitates iOS, but has awful copy. Both iOS and Android users are less likely to convert, but it attracts more iOS users.
The group as a whole has higher conversion because of the demographic shift, but every subgroup has less.
- whimsicalism 3y agoI don't follow. If one bucket has many more iOS users, it seems like you have done a bad job randomizing your treatment?
- BoppreH 3y agoIt could be self-selection happening after you randomized the groups. For example a desktop landing page advertising an app, which might be installed on either mobile operating system.