4 ms·
I read this as marketing bluff: Testing one difference guarantees that one side will beat the other (whether or not there is a true difference). Multiple test
by ronaldx 12y ago
I read this as marketing bluff:
Testing one difference guarantees that one side will beat the other (whether or not there is a true difference).
Multiple testing has a danger of the results appearing inconsistent/random (again, whether or not there is a true difference).
Encouraging people to test one difference benefits MailChimp, for exactly the reason that they say - the results will always appear to be relevant.
- gus_massa 12y agoThat’s why you must do at least a back of the envelope test to see if the results are significant. If you send two set of n mails, the expected variations is something like sqrt(n) (I don’t remember now, probably sqrt(n)/4 or sqrt(n)/2, or something like that.) Ask a statistician for the exact number. One easy experiment is to do the null A/B test. Send two sets of identical mails and analyze the difference of the result of the two sets. Usually you will get more variation that most people naively expect.