2 ms·

I am speaking in the plain language used by those of us who do science for a living. I'm using the term post hoc in the way that we in science usually do (see h

I am speaking in the plain language used by those of us who do science for a living. I'm using the term post hoc in the way that we in science usually do (see http://en.wikipedia.org/wiki/Post-hoc_analysis http://en.wikipedia.org/wiki/Post-hoc_analysis if you're not familiar). I'm not sure how high school science teachers fit into this; sounds like you had a bad experience and I'm sorry if so, but that really has nothing to do with this discussion. There is nothing wrong with "my thinking" but you appear to think so because your understanding is incorrect. Post hoc ergo propter hoc is the fallacy of thinking that, because something comes before another, that first something causes the second. I'm not talking about that at all. Post hoc analysis is something totally different - viz, it's performing non-prespecified analyses. It's more closely coupled to multiple testing than it is to the other fallacy you are referring to. It certainly is not limited to small sample sizes or sample sizes of one, in contrast to what you state. You can do science improperly with samples of any size.

_delirium 16y ago
I could be wrong, but at least in my little corner of science, this particular kind of post-hoc analysis, where you're moving thresholds on a linear scale for reporting purposes, seems not to be viewed with great suspicion, because it doesn't really produce arbitrary degrees of freedom. The worse kind of post-hoc analysis is where you can end up claiming things like "men who watched TV between 5.5 and 9 hours a week [whatever] more than men who watched TV either 1.0-2.0 or 11.5-12.5 hours/week", retroactively discovering a million probably random correlations in your data. Though even there there are ways of reducing the risk of doing so; for example, cross-validation is commonly used to test whether discovered correlations are artifacts of sampling or not, and has some pretty solid statistical theory behind it. But in this case the general hypothesis is pre-chosen and fixed, basically "men who watch TV more hours [whatever] more than men who watch TV fewer hours", and the choice of threshholds is more of a reporting step, pulling out some binned tails at either extreme for emphasis and ease of discussion, while the real correlation is full-curve-to-full-curve.
- carbocation 16y ago
  I largely agree with you. I view moving thresholds with suspicion when you're trying to report effect sizes, but sure, I mean, I don't doubt the association. I doubt the magnitude, not the association per se. And I absolutely agree that a sort of nonparametric post hoc approach that you poo-pooed would be even more highly suspect. But I don't see the choice of thresholds as simply a reporting step that can be manipulated ad lib without adjustment of your belief in the association and magnitude of effect. (I don't mean to imply that you think that you can adjust cutoffs willy-nilly - just stating my opinion clearly here.)

_delirium 16y ago

I could be wrong, but at least in my little corner of science, this particular kind of post-hoc analysis, where you're moving thresholds on a linear scale for reporting purposes, seems not to be viewed with great suspicion, because it doesn't really produce arbitrary degrees of freedom. The worse kind of post-hoc analysis is where you can end up claiming things like "men who watched TV between 5.5 and 9 hours a week [whatever] more than men who watched TV either 1.0-2.0 or 11.5-12.5 hours/week", retroactively discovering a million probably random correlations in your data. Though even there there are ways of reducing the risk of doing so; for example, cross-validation is commonly used to test whether discovered correlations are artifacts of sampling or not, and has some pretty solid statistical theory behind it. But in this case the general hypothesis is pre-chosen and fixed, basically "men who watch TV more hours [whatever] more than men who watch TV fewer hours", and the choice of threshholds is more of a reporting step, pulling out some binned tails at either extreme for emphasis and ease of discussion, while the real correlation is full-curve-to-full-curve.

carbocation 16y ago
I largely agree with you. I view moving thresholds with suspicion when you're trying to report effect sizes, but sure, I mean, I don't doubt the association. I doubt the magnitude, not the association per se. And I absolutely agree that a sort of nonparametric post hoc approach that you poo-pooed would be even more highly suspect. But I don't see the choice of thresholds as simply a reporting step that can be manipulated ad lib without adjustment of your belief in the association and magnitude of effect. (I don't mean to imply that you think that you can adjust cutoffs willy-nilly - just stating my opinion clearly here.)