5 ms·

This is a warning sign: "Men who spent more than 23 hours a week watching TV and sitting in their cars (as passengers or as drivers) had a 64 percent greater ch

This is a warning sign: "Men who spent more than 23 hours a week watching TV and sitting in their cars (as passengers or as drivers) had a 64 percent greater chance of dying from heart disease than those who sat for 11 hours a week or less." 23 hours per week vs 11 hours per week? I call this a warning sign because it screams of post-hoc analysis. Unless they then used these as pre-specified cutoffs for later studies, I would be very cautious not to overinterpret.

sdfx 16y ago
From the abstract of the study: Participants were 7744 men (20-89 yr) initially free of CVD who returned a mail-back survey during 1982. Time spent watching TV and time spent riding in a car were reported. Mortality data were ascertained through the National Death Index until December 31, 2003. They noted that they adjusted for age. I presume other factors like the economic status correlate also with the time they watched tv. Don't know if they've adjusted for that as well.
- hugh3 16y ago
  So they didn't take time spent sitting in an office into account at all? Seems odd. I spend at least two hours sitting at a desk for every hour I spend sitting on a couch or in a car seat.
  - joe_the_user 16y ago
    It's interesting indeed. Either they controlled for occupation (doubtful) or sitting in an office didn't matter or the thing whole is just off-base. One way that sitting in an office is different from watching TV involves complete passivity whereas in an office setting, your actively engaged in some task. But driving a car is probably closer to sitting in an office than watching TV. So the whole things seem problematic.
- carbocation 16y ago
  Right, but my point is not about whether or not they adjusted for the right factors. Whether or not they did that, it is clear that they took a retrospective peek at the data and said, "Aha! If we make cutpoints at 11 hours and 23 hours, we get interesting results." Adjustment for age, which should be done, has no bearing on whether this post hoc slicing of the data to give good results is proper. This is only OK so long as you're intentionally exploring, and then perhaps doing a follow-up study using those cutpoints in prespecified fashion using fresh data. In other words, if you went back and looked at your data and found the best cutoffs to maximize difference in risk, is it any surprise that you found a big difference in risk? This is why they need to validate these cutpoints in a separate study.
  - hartror 16y ago
    I'm afraid you use the post hoc fallacy incorrectly. The post hoc fallacy is drawing conclusions based on statistically insignificant data sets, normally anecdotes of size 1, at least in every day life. Now assuming this study was done by half way competent statisticians they would have followed the tried and true methods of establishing statistical significance. Sure they could have made mistakes on the way, the entire study could be completely invalidated by a detail or two. But even in that case it isn't a post hoc ergo propter hoc. Your thinking could stem from an affliction brought on by high schools teaching of science. They really only teach one of the scientific methods, that of hypothesis, experimentation, observation and conclusion in that order. However depending on the field of science and the subject at hand that isn't necessarily the order thing are done in. Science isn't a simple linear process like we are lead to believe in high school.
    - carbocation 16y ago
      I am speaking in the plain language used by those of us who do science for a living. I'm using the term post hoc in the way that we in science usually do (see http://en.wikipedia.org/wiki/Post-hoc_analysis http://en.wikipedia.org/wiki/Post-hoc_analysis if you're not familiar). I'm not sure how high school science teachers fit into this; sounds like you had a bad experience and I'm sorry if so, but that really has nothing to do with this discussion. There is nothing wrong with "my thinking" but you appear to think so because your understanding is incorrect. Post hoc ergo propter hoc is the fallacy of thinking that, because something comes before another, that first something causes the second. I'm not talking about that at all. Post hoc analysis is something totally different - viz, it's performing non-prespecified analyses. It's more closely coupled to multiple testing than it is to the other fallacy you are referring to. It certainly is not limited to small sample sizes or sample sizes of one, in contrast to what you state. You can do science improperly with samples of any size.
gjm11 16y ago
It's always wise to suspect that sort of data-mining, but if (as I think but am not sure) your point is that they look like arbitrary numbers, suggesting that they were chosen by the researchers to give a good-sounding result -- well, let's express them differently: "24 hours or more" versus "less than 12 hours". It sounds a lot more canonical that way :-). ... Except that, looking at the abstract (there's a link in the NYT article), I find this: "Men who reported >10 h/wk riding in a car or >23 h/wk of combined sedentary behavior had 82% and 64% greater risk of dying from CVD than those who reported <4 or <11 h/wk, respectively." [I've changed one unhelpful bit of notation.] So, looks like data mining after all. On the other other hand, as well as those (possibly cherry-picked) thresholds there are statements like "After age adjustment, time riding in a car and combined time spent in these two sedentary behaviors were positively (P(trend) < 0.001) associated with CVD death." which suggests something less ad hoc.
- carbocation 16y ago
  I agree with you on all points.
- phe 16y ago
  From the NYTimes article: What was unexpected was that many of the men who sat long hours and developed heart problems also exercised. Quite a few of them said they did so regularly and led active lifestyles. The men worked out, then sat in cars and in front of televisions for hours, and their risk of heart disease soared, despite the exercise. Their workouts did not counteract the ill effects of sitting. From the journal abstract: In addition, high levels of physical activity were related to notably lower rates of CVD death even in the presence of high levels of sedentary behavior. If anybody wants to look at the article, I have access to the paper.
palish 16y ago
23 hours per week vs 11 hours per week? I call this a warning sign because it screams of post-hoc analysis. Unless they then used these as pre-specified cutoffs for later studies, I would be very cautious not to overinterpret. What exactly are you saying? I honestly can't decipher the meaning of your words.
- carbocation 16y ago
  Sorry, I'm speaking a DSL I guess. I'm talking about the difference between a prespecified analysis (one that is planned in advance) and an analysis that was not planned in advance. One of the problem with analyses that are not planned in advance is that they suffer from the problems inherent in multiple testing. If I plan in advance to compare people weighing over 150 pounds to people weighing under 150 pounds, then I am conducting one test on the data. If I, instead, collect a bunch of data and run it through the ringer, torturing it until it spits out an answer that looks good to me, then I can find all kinds of spurious associations. They are more useful for hypothesis generation (i.e., generating a hypothesis that can be tested with other data). The Wikipedia article does a better job of explaining this than I have: http://en.wikipedia.org/wiki/Post-hoc_analysis http://en.wikipedia.org/wiki/Post-hoc_analysis
roundsquare 16y ago
I tend to expect that this sort of language is a bit of simplifying in the article, not necessarily an issue in the original study.

sdfx 16y ago

From the abstract of the study: Participants were 7744 men (20-89 yr) initially free of CVD who returned a mail-back survey during 1982. Time spent watching TV and time spent riding in a car were reported. Mortality data were ascertained through the National Death Index until December 31, 2003. They noted that they adjusted for age. I presume other factors like the economic status correlate also with the time they watched tv. Don't know if they've adjusted for that as well.

hugh3 16y ago
So they didn't take time spent sitting in an office into account at all? Seems odd. I spend at least two hours sitting at a desk for every hour I spend sitting on a couch or in a car seat.
- joe_the_user 16y ago
  It's interesting indeed. Either they controlled for occupation (doubtful) or sitting in an office didn't matter or the thing whole is just off-base. One way that sitting in an office is different from watching TV involves complete passivity whereas in an office setting, your actively engaged in some task. But driving a car is probably closer to sitting in an office than watching TV. So the whole things seem problematic.
carbocation 16y ago
Right, but my point is not about whether or not they adjusted for the right factors. Whether or not they did that, it is clear that they took a retrospective peek at the data and said, "Aha! If we make cutpoints at 11 hours and 23 hours, we get interesting results." Adjustment for age, which should be done, has no bearing on whether this post hoc slicing of the data to give good results is proper. This is only OK so long as you're intentionally exploring, and then perhaps doing a follow-up study using those cutpoints in prespecified fashion using fresh data. In other words, if you went back and looked at your data and found the best cutoffs to maximize difference in risk, is it any surprise that you found a big difference in risk? This is why they need to validate these cutpoints in a separate study.
- hartror 16y ago
  I'm afraid you use the post hoc fallacy incorrectly. The post hoc fallacy is drawing conclusions based on statistically insignificant data sets, normally anecdotes of size 1, at least in every day life. Now assuming this study was done by half way competent statisticians they would have followed the tried and true methods of establishing statistical significance. Sure they could have made mistakes on the way, the entire study could be completely invalidated by a detail or two. But even in that case it isn't a post hoc ergo propter hoc. Your thinking could stem from an affliction brought on by high schools teaching of science. They really only teach one of the scientific methods, that of hypothesis, experimentation, observation and conclusion in that order. However depending on the field of science and the subject at hand that isn't necessarily the order thing are done in. Science isn't a simple linear process like we are lead to believe in high school.
  - carbocation 16y ago
    I am speaking in the plain language used by those of us who do science for a living. I'm using the term post hoc in the way that we in science usually do (see http://en.wikipedia.org/wiki/Post-hoc_analysis http://en.wikipedia.org/wiki/Post-hoc_analysis if you're not familiar). I'm not sure how high school science teachers fit into this; sounds like you had a bad experience and I'm sorry if so, but that really has nothing to do with this discussion. There is nothing wrong with "my thinking" but you appear to think so because your understanding is incorrect. Post hoc ergo propter hoc is the fallacy of thinking that, because something comes before another, that first something causes the second. I'm not talking about that at all. Post hoc analysis is something totally different - viz, it's performing non-prespecified analyses. It's more closely coupled to multiple testing than it is to the other fallacy you are referring to. It certainly is not limited to small sample sizes or sample sizes of one, in contrast to what you state. You can do science improperly with samples of any size.

gjm11 16y ago

It's always wise to suspect that sort of data-mining, but if (as I think but am not sure) your point is that they look like arbitrary numbers, suggesting that they were chosen by the researchers to give a good-sounding result -- well, let's express them differently: "24 hours or more" versus "less than 12 hours". It sounds a lot more canonical that way :-). ... Except that, looking at the abstract (there's a link in the NYT article), I find this: "Men who reported >10 h/wk riding in a car or >23 h/wk of combined sedentary behavior had 82% and 64% greater risk of dying from CVD than those who reported <4 or <11 h/wk, respectively." [I've changed one unhelpful bit of notation.] So, looks like data mining after all. On the other other hand, as well as those (possibly cherry-picked) thresholds there are statements like "After age adjustment, time riding in a car and combined time spent in these two sedentary behaviors were positively (P(trend) < 0.001) associated with CVD death." which suggests something less ad hoc.

carbocation 16y ago
I agree with you on all points.
phe 16y ago
From the NYTimes article: What was unexpected was that many of the men who sat long hours and developed heart problems also exercised. Quite a few of them said they did so regularly and led active lifestyles. The men worked out, then sat in cars and in front of televisions for hours, and their risk of heart disease soared, despite the exercise. Their workouts did not counteract the ill effects of sitting. From the journal abstract: In addition, high levels of physical activity were related to notably lower rates of CVD death even in the presence of high levels of sedentary behavior. If anybody wants to look at the article, I have access to the paper.

palish 16y ago

23 hours per week vs 11 hours per week? I call this a warning sign because it screams of post-hoc analysis. Unless they then used these as pre-specified cutoffs for later studies, I would be very cautious not to overinterpret. What exactly are you saying? I honestly can't decipher the meaning of your words.

carbocation 16y ago
Sorry, I'm speaking a DSL I guess. I'm talking about the difference between a prespecified analysis (one that is planned in advance) and an analysis that was not planned in advance. One of the problem with analyses that are not planned in advance is that they suffer from the problems inherent in multiple testing. If I plan in advance to compare people weighing over 150 pounds to people weighing under 150 pounds, then I am conducting one test on the data. If I, instead, collect a bunch of data and run it through the ringer, torturing it until it spits out an answer that looks good to me, then I can find all kinds of spurious associations. They are more useful for hypothesis generation (i.e., generating a hypothesis that can be tested with other data). The Wikipedia article does a better job of explaining this than I have: http://en.wikipedia.org/wiki/Post-hoc_analysis http://en.wikipedia.org/wiki/Post-hoc_analysis

roundsquare 16y ago

I tend to expect that this sort of language is a bit of simplifying in the article, not necessarily an issue in the original study.