>"there's a .04 probability that there's no effect and the results occurred by chance." No, the p value is the probability of observing a result at least as ex

>"there's a .04 probability that there's no effect and the results occurred by chance." No, the p value is the probability of observing a result at least as extreme as your own given the null hypothesis is true. There are two errors here 1) Transposing the conditional: P(A|B) != P(B|A) http://rationalwiki.org/wiki/Confusion_of_the_inverse http://rationalwiki.org/wiki/Confusion_of_the_inverse 2) Deviations from the null hypothesis can occur even the absence of a treatment effect, ie one of your model assumptions is wrong, baseline differences, etc. Anyway, that is why instead of statistical significance researchers need to estimate the size of the effect. If you estimate the effect is in the range -1 to +2 then you publish that result. Others also estimate the range and see if these are consistent with each other.