3 ms·
"We recommend testing only one difference between the A and B groups. When there are several differences between test groups, it’s difficult to figure out which
by mfrommil 12y ago
"We recommend testing only one difference between the A and B groups. When there are several differences between test groups, it’s difficult to figure out which change impacted your engagement."
A/B testing may be better to figure out one specific aspect (time to send, sender name) but wouldn't multivariate testing make sense if you were looking for the best combination of those variables? i.e. if I wanted to find out that sending an email at 9pm, from a corporate account, with larger images is the best combination?
I'm interested to hear others' opinions on the pros/cons.
- ronaldx 12y agoI read this as marketing bluff: Testing one difference guarantees that one side will beat the other (whether or not there is a true difference). Multiple testing has a danger of the results appearing inconsistent/random (again, whether or not there is a true difference). Encouraging people to test one difference benefits MailChimp, for exactly the reason that they say - the results will always appear to be relevant.
- gus_massa 12y agoThat’s why you must do at least a back of the envelope test to see if the results are significant. If you send two set of n mails, the expected variations is something like sqrt(n) (I don’t remember now, probably sqrt(n)/4 or sqrt(n)/2, or something like that.) Ask a statistician for the exact number. One easy experiment is to do the null A/B test. Send two sets of identical mails and analyze the difference of the result of the two sets. Usually you will get more variation that most people naively expect.