3 ms·

I'm afraid you use the post hoc fallacy incorrectly. The post hoc fallacy is drawing conclusions based on statistically insignificant data sets, normally anecdo

by hartror 16y ago

I'm afraid you use the post hoc fallacy incorrectly. The post hoc fallacy is drawing conclusions based on statistically insignificant data sets, normally anecdotes of size 1, at least in every day life. Now assuming this study was done by half way competent statisticians they would have followed the tried and true methods of establishing statistical significance. Sure they could have made mistakes on the way, the entire study could be completely invalidated by a detail or two. But even in that case it isn't a post hoc ergo propter hoc. Your thinking could stem from an affliction brought on by high schools teaching of science. They really only teach one of the scientific methods, that of hypothesis, experimentation, observation and conclusion in that order. However depending on the field of science and the subject at hand that isn't necessarily the order thing are done in. Science isn't a simple linear process like we are lead to believe in high school.

carbocation 16y ago
I am speaking in the plain language used by those of us who do science for a living. I'm using the term post hoc in the way that we in science usually do (see http://en.wikipedia.org/wiki/Post-hoc_analysis http://en.wikipedia.org/wiki/Post-hoc_analysis if you're not familiar). I'm not sure how high school science teachers fit into this; sounds like you had a bad experience and I'm sorry if so, but that really has nothing to do with this discussion. There is nothing wrong with "my thinking" but you appear to think so because your understanding is incorrect. Post hoc ergo propter hoc is the fallacy of thinking that, because something comes before another, that first something causes the second. I'm not talking about that at all. Post hoc analysis is something totally different - viz, it's performing non-prespecified analyses. It's more closely coupled to multiple testing than it is to the other fallacy you are referring to. It certainly is not limited to small sample sizes or sample sizes of one, in contrast to what you state. You can do science improperly with samples of any size.
- _delirium 16y ago
  I could be wrong, but at least in my little corner of science, this particular kind of post-hoc analysis, where you're moving thresholds on a linear scale for reporting purposes, seems not to be viewed with great suspicion, because it doesn't really produce arbitrary degrees of freedom. The worse kind of post-hoc analysis is where you can end up claiming things like "men who watched TV between 5.5 and 9 hours a week [whatever] more than men who watched TV either 1.0-2.0 or 11.5-12.5 hours/week", retroactively discovering a million probably random correlations in your data. Though even there there are ways of reducing the risk of doing so; for example, cross-validation is commonly used to test whether discovered correlations are artifacts of sampling or not, and has some pretty solid statistical theory behind it. But in this case the general hypothesis is pre-chosen and fixed, basically "men who watch TV more hours [whatever] more than men who watch TV fewer hours", and the choice of threshholds is more of a reporting step, pulling out some binned tails at either extreme for emphasis and ease of discussion, while the real correlation is full-curve-to-full-curve.
  - carbocation 16y ago
    I largely agree with you. I view moving thresholds with suspicion when you're trying to report effect sizes, but sure, I mean, I don't doubt the association. I doubt the magnitude, not the association per se. And I absolutely agree that a sort of nonparametric post hoc approach that you poo-pooed would be even more highly suspect. But I don't see the choice of thresholds as simply a reporting step that can be manipulated ad lib without adjustment of your belief in the association and magnitude of effect. (I don't mean to imply that you think that you can adjust cutoffs willy-nilly - just stating my opinion clearly here.)

by hartror 16y ago

carbocation 16y ago

I am speaking in the plain language used by those of us who do science for a living. I'm using the term post hoc in the way that we in science usually do (see http://en.wikipedia.org/wiki/Post-hoc_analysis http://en.wikipedia.org/wiki/Post-hoc_analysis if you're not familiar). I'm not sure how high school science teachers fit into this; sounds like you had a bad experience and I'm sorry if so, but that really has nothing to do with this discussion. There is nothing wrong with "my thinking" but you appear to think so because your understanding is incorrect. Post hoc ergo propter hoc is the fallacy of thinking that, because something comes before another, that first something causes the second. I'm not talking about that at all. Post hoc analysis is something totally different - viz, it's performing non-prespecified analyses. It's more closely coupled to multiple testing than it is to the other fallacy you are referring to. It certainly is not limited to small sample sizes or sample sizes of one, in contrast to what you state. You can do science improperly with samples of any size.

_delirium 16y ago
I could be wrong, but at least in my little corner of science, this particular kind of post-hoc analysis, where you're moving thresholds on a linear scale for reporting purposes, seems not to be viewed with great suspicion, because it doesn't really produce arbitrary degrees of freedom. The worse kind of post-hoc analysis is where you can end up claiming things like "men who watched TV between 5.5 and 9 hours a week [whatever] more than men who watched TV either 1.0-2.0 or 11.5-12.5 hours/week", retroactively discovering a million probably random correlations in your data. Though even there there are ways of reducing the risk of doing so; for example, cross-validation is commonly used to test whether discovered correlations are artifacts of sampling or not, and has some pretty solid statistical theory behind it. But in this case the general hypothesis is pre-chosen and fixed, basically "men who watch TV more hours [whatever] more than men who watch TV fewer hours", and the choice of threshholds is more of a reporting step, pulling out some binned tails at either extreme for emphasis and ease of discussion, while the real correlation is full-curve-to-full-curve.
- carbocation 16y ago
  I largely agree with you. I view moving thresholds with suspicion when you're trying to report effect sizes, but sure, I mean, I don't doubt the association. I doubt the magnitude, not the association per se. And I absolutely agree that a sort of nonparametric post hoc approach that you poo-pooed would be even more highly suspect. But I don't see the choice of thresholds as simply a reporting step that can be manipulated ad lib without adjustment of your belief in the association and magnitude of effect. (I don't mean to imply that you think that you can adjust cutoffs willy-nilly - just stating my opinion clearly here.)