3 ms·
It doubled, or in this case, increased by 1.6%. That's nice, but how can you know that a 1.6% increase is a direct result of these tweaks, instead of just natu
by acabal 14y ago
It doubled, or in this case, increased by 1.6%. That's nice, but how can you know that a 1.6% increase is a direct result of these tweaks, instead of just natural ups and downs in email response? Maybe 1.6% of recipients were having an unnaturally good day and decided to click that month? That's only 1 in 98 people after all. If it had been 10% or something like that then that's an obvious difference, but 1.6%?
- wolffnc3 14y agoTypically A/B tests are run until they reach a statistically significant result. The article doesn't include the sample size, but with a sample size as small as 500 users for each test an increase from 1.6% to 3.2% would be a change larger than 1 standard deviation, and thus can be assumed to be more than just random noise.