3 ms·
They could be more certain with a multivariate test. I count 2^7 = 64 possible variations (counting the madlib/form as just one point of difference), which wou
by patio11 17y ago
They could be more certain with a multivariate test. I count 2^7 = 64 possible variations (counting the madlib/form as just one point of difference), which would take an awful lot of traffic to discriminate between. So I sort of kind of understand the desire to wrap it up in one simple A/B test.
But you're right, of course, there is no reason to assume that the change was from the obvious of the 7 changes instead of from all (or one!) of the other six changes. Did anybody notice that the button call to action changed? I've seen A/B tests where that did 20% by itself. Did anybody notice that the header call to action changed? I've seen A/B tests where that did 20% by itself. Did anybody notice that they added the words "Thank you."? I've seen A/B tests where... actually, I haven't for "Thank you." (if you've got one, please share) But you get the general idea.
- eru 17y ago> I count 2^7 = 64 possible variations (counting the madlib/form as just one point of difference), which would take an awful lot of traffic to discriminate between. A fractional factorial design (http://en.wikipedia.org/wiki/Fractional_factorial_design http://en.wikipedia.org/wiki/Fractional_factorial_design) of the experiments and other statistic techniques (http://en.wikipedia.org/wiki/Bayesian_experimental_design http://en.wikipedia.org/wiki/Bayesian_experimental_design) may decrease the amount of traffic needed.