3 ms·

Although these results are not exactly impressive or compelling (they don't make me want to change my exercise habits), it's reassuring to see researchers go th

by bnjmn 6y ago

Although these results are not exactly impressive or compelling (they don't make me want to change my exercise habits), it's reassuring to see researchers go through with publishing underwhelming results, rather than cherry-picking only the interesting results and sitting on the rest, which is a major contributing factor to the crisis of confidence/replication in the social sciences. - https://en.wikipedia.org/wiki/Cherry_picking https://en.wikipedia.org/wiki/Cherry_picking - https://www.nature.com/news/scientific-method-statistical-errors-1.14700 https://www.nature.com/news/scientific-method-statistical-er... - https://fivethirtyeight.com/features/science-isnt-broken/ https://fivethirtyeight.com/features/science-isnt-broken/ - etc...

tomerico 6y ago
Why do you say that the results aren't impressive? 49% reduction in all cause mortality when High intensity interval training is compared to moderate intensity continuous training seems like a very strong result.
- ardit33 6y ago
  "an absolute risk reduction of 1.7 percentage points was observed after HIIT (hazard ratio 0.63, 95% confidence interval 0.33 to 1.20) and an absolute increased risk of 1.2 percentage points after MICT (1.24, 0.73 to 2.10)." So, 1.7% vs 1.2% seems just above noise level. The OP is right, the numbers are not that impressive. We are not talking about reducing overall mortality by double digits... Summary: looks like exercising is good, and adding HIIT has an increasing effect of reducing mortality, but the overall effect is small.
  - oarabbus_ 6y ago
    >Participants were randomised to two sessions weekly of high intensity interval training at about 90% of peak heart rate (HIIT, n=400), moderate intensity continuous training at about 70% of peak heart rate (MICT, n=387), or to follow the national guidelines for physical activity (n=780; control group); all for five years. I can't quite tell by your summary whether you are saying "exercising is good, but the overall effect on mortality is small", or "exercising is good, and the effect of adding HIIT to baseline exercise is small". You cannot make the former statement from this study as control group were not non-exercisers (and adherence was decent). The latter statement, does seem to be supported. Regarding exercise in general, the literature shows the mortality gap between exercisers and non-exercisers is absolutely massive.
  - SpaceNugget 6y ago
    2.9% seems like a significat risk reduction to me. HIIT results in a 1.7% risk REDUCTION and MICT results in a 1.2% risk INCREASE.
  - tomtheelder 6y ago
    Those are absolute risk reductions, not relative ones. The 1.7 percentage point reduction equates to a 37% reduction in all cause mortality, and a 1.2 percentage point increase is a 25% increase. Those are huge numbers. The problem is that the confidence interval is really wide.
  - BenoitEssiambre 6y ago
    >but the overall effect is small. quibble: non significant, wide confidence doesn't mean "small", it means _unknown_. It means the data is too sparse and/or too noisy to tell.
    - pas 6y ago
      Doesn't this also imply that the effect is "small" even if real? After all if there would be a very strong correlation (eg a true and exclusive casual chain) then we would see a huge signal even in small and noisy datasets. Or am I missing something?
      - BenoitEssiambre 6y ago
        No. If you look closely at the data you might be able to draw such conclusions but lack of statistical significance often doesn't suggest or imply a small effect. Notice that in this case in particular, the confidence intervals are consistent with very large positive or fairly large negative effects. Don't underestimate the amount of noise often found in studies. Lack of significance usually just means that data is too noisy to tell us anything. If you get significance you get to say: the data is probably not pure noise but the effect could still be very tiny or caused by systematic measurement errors. Null hypothesis testing is pretty useless really.
        pas 6y ago
        Thanks for your detailed reply! I meant that even this dataset puts a limit on the effect size, if viewed as an "evidence of absence of clear and large effect". Of course since "everything is correlated" [0] expecting such truly simple signals might be nonsensical/pointless. I was just lamenting the lack of simple magical treatments basically. [0] https://www.gwern.net/Everything https://www.gwern.net/Everything
        BenoitEssiambre 6y ago
        That gwern page is excellent!
- unreal37 6y ago
  No difference between people managing their own exercise program compared with those who had a trainer. "These differences were not statistically significant" according to the study. The study itself says that they observed no difference. That is not a strong result. Which is OK.
- bnjmn 6y ago
  Mostly because they admit those results are not statistically significant (for example, at the bottom of the summary diagram). That said, I admire their honesty, and I hope these results suggest other kinds of experiments to other researchers. Maybe the study just needs to be larger, or longer. Focusing on specific kinds of mortality might also give stronger results than measuring "all cause" mortality. I say that because it seems like cancer was the biggest killer in these groups. I am not a doctor, but I wouldn't have thought cancer was causally related to (lack of) exercise, the way cardiovascular diseases are believed to be. I'd be interested to see larger studies with enough non-cancer deaths to say something statistically significant about the effect of exercise on those outcomes.
  - travisoneill1 6y ago
    Why even bother doing a study with a sample size so small that a difference as large as 49% is still not significant?
    - TJSomething 6y ago
      Because running a large enough study is expensive and it's cost-effective to perform a smaller study to determine an approximate size of the effect. Then, you can design the next study to have enough power to distinguish the expected effect size from the null hypothesis. Additionally, these results could be aggregated with comparable results to yield a stronger result.
    - andor 6y ago
      They had 1567 participants. Compared to control (HIIT-like), HIIT reduced risk by 1.7% while moderate intensity training increased it by 1.2%. The overall mortality in the control group was 4.7%. For a controlled trial, that seems like a large number of participants (correct me if I'm wrong), they just were too healthy ;-) I assume the significance problem is inherent to studies looking at mortality as the outcome is very binary and can take a long time to manifest. As an effect on all-cause mortality can be seen as the ultimate metric of how healthy something is, it's probably still worth it to investigate. In this case, they made it quite difficult for themselves by comparing active people to other active people. It's also possible that they made other observations during the study that are or will be published separately.

by bnjmn 6y ago

tomerico 6y ago

Why do you say that the results aren't impressive? 49% reduction in all cause mortality when High intensity interval training is compared to moderate intensity continuous training seems like a very strong result.

ardit33 6y ago
"an absolute risk reduction of 1.7 percentage points was observed after HIIT (hazard ratio 0.63, 95% confidence interval 0.33 to 1.20) and an absolute increased risk of 1.2 percentage points after MICT (1.24, 0.73 to 2.10)." So, 1.7% vs 1.2% seems just above noise level. The OP is right, the numbers are not that impressive. We are not talking about reducing overall mortality by double digits... Summary: looks like exercising is good, and adding HIIT has an increasing effect of reducing mortality, but the overall effect is small.
- oarabbus_ 6y ago
  >Participants were randomised to two sessions weekly of high intensity interval training at about 90% of peak heart rate (HIIT, n=400), moderate intensity continuous training at about 70% of peak heart rate (MICT, n=387), or to follow the national guidelines for physical activity (n=780; control group); all for five years. I can't quite tell by your summary whether you are saying "exercising is good, but the overall effect on mortality is small", or "exercising is good, and the effect of adding HIIT to baseline exercise is small". You cannot make the former statement from this study as control group were not non-exercisers (and adherence was decent). The latter statement, does seem to be supported. Regarding exercise in general, the literature shows the mortality gap between exercisers and non-exercisers is absolutely massive.
- SpaceNugget 6y ago
  2.9% seems like a significat risk reduction to me. HIIT results in a 1.7% risk REDUCTION and MICT results in a 1.2% risk INCREASE.
- tomtheelder 6y ago
  Those are absolute risk reductions, not relative ones. The 1.7 percentage point reduction equates to a 37% reduction in all cause mortality, and a 1.2 percentage point increase is a 25% increase. Those are huge numbers. The problem is that the confidence interval is really wide.
- BenoitEssiambre 6y ago
  >but the overall effect is small. quibble: non significant, wide confidence doesn't mean "small", it means _unknown_. It means the data is too sparse and/or too noisy to tell.
  - pas 6y ago
    Doesn't this also imply that the effect is "small" even if real? After all if there would be a very strong correlation (eg a true and exclusive casual chain) then we would see a huge signal even in small and noisy datasets. Or am I missing something?
    - BenoitEssiambre 6y ago
      No. If you look closely at the data you might be able to draw such conclusions but lack of statistical significance often doesn't suggest or imply a small effect. Notice that in this case in particular, the confidence intervals are consistent with very large positive or fairly large negative effects. Don't underestimate the amount of noise often found in studies. Lack of significance usually just means that data is too noisy to tell us anything. If you get significance you get to say: the data is probably not pure noise but the effect could still be very tiny or caused by systematic measurement errors. Null hypothesis testing is pretty useless really.
      - pas 6y ago
        Thanks for your detailed reply! I meant that even this dataset puts a limit on the effect size, if viewed as an "evidence of absence of clear and large effect". Of course since "everything is correlated" [0] expecting such truly simple signals might be nonsensical/pointless. I was just lamenting the lack of simple magical treatments basically. [0] https://www.gwern.net/Everything https://www.gwern.net/Everything
        BenoitEssiambre 6y ago
        That gwern page is excellent!
unreal37 6y ago
No difference between people managing their own exercise program compared with those who had a trainer. "These differences were not statistically significant" according to the study. The study itself says that they observed no difference. That is not a strong result. Which is OK.
bnjmn 6y ago
Mostly because they admit those results are not statistically significant (for example, at the bottom of the summary diagram). That said, I admire their honesty, and I hope these results suggest other kinds of experiments to other researchers. Maybe the study just needs to be larger, or longer. Focusing on specific kinds of mortality might also give stronger results than measuring "all cause" mortality. I say that because it seems like cancer was the biggest killer in these groups. I am not a doctor, but I wouldn't have thought cancer was causally related to (lack of) exercise, the way cardiovascular diseases are believed to be. I'd be interested to see larger studies with enough non-cancer deaths to say something statistically significant about the effect of exercise on those outcomes.
- travisoneill1 6y ago
  Why even bother doing a study with a sample size so small that a difference as large as 49% is still not significant?
  - TJSomething 6y ago
    Because running a large enough study is expensive and it's cost-effective to perform a smaller study to determine an approximate size of the effect. Then, you can design the next study to have enough power to distinguish the expected effect size from the null hypothesis. Additionally, these results could be aggregated with comparable results to yield a stronger result.
  - andor 6y ago
    They had 1567 participants. Compared to control (HIIT-like), HIIT reduced risk by 1.7% while moderate intensity training increased it by 1.2%. The overall mortality in the control group was 4.7%. For a controlled trial, that seems like a large number of participants (correct me if I'm wrong), they just were too healthy ;-) I assume the significance problem is inherent to studies looking at mortality as the outcome is very binary and can take a long time to manifest. As an effect on all-cause mortality can be seen as the ultimate metric of how healthy something is, it's probably still worth it to investigate. In this case, they made it quite difficult for themselves by comparing active people to other active people. It's also possible that they made other observations during the study that are or will be published separately.