3 ms·
The short version of the error: If in your data you find that A has no statistically significant effect, but B does have a statistically significant effect, thi
by _delirium 15y ago
The short version of the error: If in your data you find that A has no statistically significant effect, but B does have a statistically significant effect, this does not automatically show that B has, with statistical significance, more effect than A. To do that you have to do a statistical test on the difference in the effects.
I believe (?) this is normally done correctly in medical studies with placebos, where the typical analysis is to show that a drug has a statistically significant effect compared to the placebo baseline; it's not sufficient to show that the drug has an effect compared to a no-drug baseline, and, separately, that a placebo doesn't.
- markkat 15y agoIt's true. Many anti-depressant drugs have no significant effect when compared to placebo. http://www.plosmedicine.org/article/info%3Adoi%2F10.1371%2Fjournal.pmed.0050045 http://www.plosmedicine.org/article/info%3Adoi%2F10.1371%2Fj...
- _delirium 15y agoI don't believe the controversy over that is whether the statistics were done properly, but over: 1) replicability, in particular whether significance is still found when doing large-scale meta-analyses; and 2) whether blindness of the studies is compromised by "unblinding" effects where a drug's side-effects can tip off the doctor or patient about whether they got a placebo or not. But it's been a while since I waded into that flamewar of a debate...
- ceol 15y agoNo, according to this one controversial[0] study, anti-depressant drugs have no statistical significance compared to placebo except in cases of severe depression. [0]: http://www.plosmedicine.org/article/comments/info%3Adoi%2F10.1371%2Fjournal.pmed.0050045 http://www.plosmedicine.org/article/comments/info%3Adoi%2F10...
- JoachimSchipper 15y agoYou may be simplifying too much to be understood. In medical terms, if your drug is significantly more effective than doing nothing, but a placebo is not significantly more effective than doing nothing, your drug need not be significantly more effective than the placebo. (Yes, medicine typically avoids this particular error.) (E.g. if doing nothing cures 10%, the placebo cures 11% and the drug cures 12%, only the difference drug/nothing may be significant.)
- deleted 15y ago[deleted]
- starwed 15y agoOr: if I compare three different distributions, it's possible for A&C to be distinct, but for A&B and B&C to overlap. As the linked article says, the idea of statistical significance being a binary state is to blame here.
- disgruntledphd2 15y agoA good paper on this error is from Andrew Gelman www.stat.columbia.edu/~gelman/research/unpublished/signif3.pdf
- lucisferre 15y agoI'm curious about the example in the article. If the effects were measured at 30% and 15%, which are the change in some measured norm, then isn't the difference in differences actually 100% not 15%?
- _delirium 15y agoOften you still use units of % change relative to the initial quantity, though I suppose you could define it either way. For example, if we're comparing a country with an inflation rate of 4% and one with an inflation of 8%, we might say that the difference between the rates is 4%, in that |rate1-rate2| = 0.04, as opposed to measuring one in terms of percent of the other.
- lucisferre 15y agoI'm asking because the author talks about the "difference in differences" and as early as highschool physics they drill into your head that the difference between two things is represented by the percentage difference not it's absolute difference. Saying 15% is negligible when it is actually a 100% change in the effect seems incorrect somehow.