3 ms·

The t-test is invalid because duration data is non-normal. You can actually see duration data as a histogram in GA if you want to test hypotheses on it's distri

The t-test is invalid because duration data is non-normal. You can actually see duration data as a histogram in GA if you want to test hypotheses on it's distribution.

btilly 13y ago
I came here to say this, and am glad someone else said it first. That on top of the unwarranted distribution assumption is very dangerous. How dangerous? If the true distribution has heavier tails than assumed, then the true variance can be many times what is assumed. If the true variance is, say, 4x what is assumed then an average standard deviation variation will show up as passing a 95% confidence interval in a random direction. Getting to 99% confidence by accident becomes quite easy, with bad results.
- tel 13y ago
  Ugh. I didn't even read the article and was looking for this response. Whenever I see "A/B test" I am usually right in assuming that the author is about to assume normality.
  - btilly 13y ago
    I understand, but the author of this particular article actually does know better and usually gets his statistics right.
- jimmybot 13y ago
  Can you be more specific on why you think the t-test is invalid when the data is non-normal? Duration data may be non-normal, but given finite visits and a reasonable number of visits, sampling for the mean surely is.
  - btilly 13y ago
    You are right that in the presence of sufficient data, everything converges to the normal and we're fine. However the difference between the t-test and the normal to which it is an approximation is that the t-test takes into account corrections the rate of convergence to the estimated normal. The details of that convergence is dependent upon the distribution of individual samples, and the t-test is predicated on the assumption that each and every sample is from a normal distribution.
saosebastiao 13y ago
Not only that, but it appears the sample sizes are not equivalent.
bobbinsIII 13y ago
you can still use a t-test for non-normal data...

The t-test is invalid because duration data is non-normal. You can actually see duration data as a histogram in GA if you want to test hypotheses on it's distribution.

btilly 13y ago

I came here to say this, and am glad someone else said it first. That on top of the unwarranted distribution assumption is very dangerous. How dangerous? If the true distribution has heavier tails than assumed, then the true variance can be many times what is assumed. If the true variance is, say, 4x what is assumed then an average standard deviation variation will show up as passing a 95% confidence interval in a random direction. Getting to 99% confidence by accident becomes quite easy, with bad results.

tel 13y ago
Ugh. I didn't even read the article and was looking for this response. Whenever I see "A/B test" I am usually right in assuming that the author is about to assume normality.
- btilly 13y ago
  I understand, but the author of this particular article actually does know better and usually gets his statistics right.
jimmybot 13y ago
Can you be more specific on why you think the t-test is invalid when the data is non-normal? Duration data may be non-normal, but given finite visits and a reasonable number of visits, sampling for the mean surely is.
- btilly 13y ago
  You are right that in the presence of sufficient data, everything converges to the normal and we're fine. However the difference between the t-test and the normal to which it is an approximation is that the t-test takes into account corrections the rate of convergence to the estimated normal. The details of that convergence is dependent upon the distribution of individual samples, and the t-test is predicated on the assumption that each and every sample is from a normal distribution.

saosebastiao 13y ago

Not only that, but it appears the sample sizes are not equivalent.

bobbinsIII 13y ago

you can still use a t-test for non-normal data...