3 ms·
Finally! It looks like someone actually did some simple math to determine if their results were statistically meaningful. I wish more people would include a sec
by CytokineStorm 17y ago
Finally! It looks like someone actually did some simple math to determine if their results were statistically meaningful. I wish more people would include a section like this when they talk about conversion rates.
- jjvens 17y agoIt also helps to do the right test. The sampling distribution of the difference between percentages/proportions is a normal distribution, not a Student's t-distribution (that one is used for means, not percentages). The z-value for the difference is 91.86, which makes the p-value about 0. This means that, when there's 300,000 people in your sample, there is no need for statistical testing!
- sesqu 17y ago15% looks great, but how confident are we? Running a one-sided Student’s t-test with a means difference of 14%, our experimental data yields a P value of 0.000051. This means that there is only a 0.0051% chance that we would obtain a 14% (or greater) improvement if the real effect wasn’t at least this large. Err... Student's t doesn't assign a probability to P(measured>=14 | actual<14), it assigns one to P(measured>=14 | actual=14) under the assumption of equal variances (which here means actual=0, so clearly not applicable). I figure that the actual effect is 16.05% +- 0.17%. And that's simply by assuming 145k is close enough to infinity, and for IE.html only. That is, unless I'm mistaken about this, since I grok only the fairly basic stuff. Which is certainly plausible in and of itself.