3 ms·
Ooh, I think you gave me a good idea for a blog topic. 1) You can run multiple A/B tests at the same time without compromising results, if you assume they are
by patio11 17y ago
Ooh, I think you gave me a good idea for a blog topic.
1) You can run multiple A/B tests at the same time without compromising results, if you assume they are independent. Your stats teachers at college will be mad at me for this, but "That is a safe assumption 99% of the time." (Shh!)
2) In terms of prioritizing your time, I wouldn't worry about things you think are likely to be marginal unless they're at key areas in your application where a 1% marginal lift would matter to the business. For example: 1% improvement in observed results of the font selection dialog box: worthless to the business. 1% improvement in the shopping cart: heck yes worth it to the business.
3) We often underestimate how effective certain changes are going to be. If you throw up a test and get statistically insignificant results, that doesn't cost you anything and you still learn something: oh well, learned that users do not empirically think the difference between A and B is supremely compelling. If on the other hand you say "Eh, this is likely to only be a 1% factor" and it turns out to be a 10% factor ("BUTTON TEXT?! WHO CARES ABOUT FREAKING BUTTON TEXT?! TEN FREAKING PERCENT?!"), you'll get statistically significant results and learn very valuable things.
- bemmu 17y agoI'll be honored if I actually managed to inspire the great patio11 to write a new post :) And yes, I did learn that the thing I was testing doesn't have a very large effect, which was valuable to know.