6 ms·

This report triggered several of my alarm bells, so I investigated further. I had a look at the quoted study. "RESULTS: No differences between trials ( P > 0.0

This report triggered several of my alarm bells, so I investigated further. I had a look at the quoted study. "RESULTS: No differences between trials ( P > 0.05) were found in the overall plasma triglyceride, glucose, or insulin responses during the HFGTT. " It takes some balls to dismiss a study based on one sentence in the abstract, but this is an admission of a common and fatal mistake, so I'm going to do exactly that. They're claiming there is no difference between groups because they were unable to show that, if there was an inherent difference, the chance they'd see these results is at most 5%. There's an easy way to obtain such a result: small sample size. And, indeed, they only compared 2 groups of 5 people. There is a proper way to show two groups are the same, and it's called "equivalence testing." It has the same kind of statistical requirements that normal testing does. What they did instead is totally statically invalid. There's not a soft way to put it: this study proves nothing. Let it be an example in training your defenses against bad science.

glastra 8y ago
The results themselves are not wrong per se. Reaching this conclusion from those results, however, is frankly insulting and should make all authors of the paper ashamed. > These data indicate that physical inactivity (e.g., sitting ~13.5 hrs/day and <4,000 steps/day) creates a condition whereby people become "resistant" to the metabolic improvements that are typically derived from an acute bout of aerobic exercise (i.e., exercise resistance). Also note that the researchers _immediately_ re-fed the group who participated in the intense exercise: > The additional energy expenditure from the exercise session was estimated via indirect calorimetry. This energy was then replaced with additional caloric intake with the post-exercise dinner (Clif® Builder’s®; ~40%/30%/30% of calories from carbohydrate, fat, and protein, respectively). This is especially bad because it isn't mentioned anywhere in the abstract (I checked the full-text). Shit science.
cjslep 8y ago
I also wanted to dig into this. The null hypothesis and burden-of-proofs change between p tests and equivalence testing, right? P tests: null hypothesis is that there is nothing different, burden of proof to suggest there is a difference. Equivalence testing: null hypothesis is that there is something different, burden of proof to suggest there is no difference. But maybe this is just a misapplied p-test? So while the "RESULTS" section of the paper is fine: No differences between trials ( P > 0.05) were found in the overall plasma triglyceride, glucose, or insulin responses during the HFGTT. The "CONCLUSION" section is where they invert the statistics of their own study and literally suggest a p-test-able difference despite their results section literally concluding a p-test-able non-difference: These data indicate that physical inactivity (e.g., sitting ~13.5 hrs/day and <4,000 steps/day) creates a condition whereby people become "resistant" to the metabolic improvements that are typically derived from an acute bout of aerobic exercise (i.e., exercise resistance).
- Darmani 8y ago
  Yes, they used a test designed to find a difference between two groups, and it failed to report a difference. Your analysis of that is correct, although the terminology "p-test" to describe "a statistical test designed to find a difference" is not. I don't know the general term for this class of tests. The definition of p-values is a bit wonky (there's actually a set of null hypotheses rather than a single one, and it involves a supremum over that set), but the line "if we're wrong, then the chance of seeing this data is at most p" is pretty accurate. All kinds of frequentist statistical tests use them, including equivalence tests.
photonios 8y ago
Excellent analysis! For those interested, here's the study the article quotes: https://www.ncbi.nlm.nih.gov/pubmed/30763169 https://www.ncbi.nlm.nih.gov/pubmed/30763169
anjc 8y ago
Can you explain this common mistake further? > They're claiming there is no difference between groups because they were unable to show that, if there was an inherent difference, the chance they'd see these results is at most 5%. It seems to me that they're claiming that they failed to reject the null hypothesis, the end. I.e. according to this statistic, there was no significant difference in the means between all groups in the ANOVA. This was a small sample size, but the analysis seems to include multiple trials for each participant. This seems like a standard use of ANOVA to me. So how can you say that the alpha value is wrong just based on this one sentence from the abstract? It seems to me that you'd have to know the domain, know what effect to expect, and so on
- glastra 8y ago
  Not OP, but [0]: > If the data do not contradict the null hypothesis, then only a weak conclusion can be made: namely, that the observed data set provides no strong evidence against the null hypothesis. In this case, because the null hypothesis could be true or false, in some contexts this is interpreted as meaning that the data give insufficient evidence to make any conclusion; in other contexts it is interpreted as meaning that there is no evidence to support changing from a currently useful regime to a different one. Also, keep in mind that in this study, the intervention (the difference between both groups) was, basically: perform an intense 1-hour bout of exercise and immediately replenish all calories spent. The null hypothesis is, thus: "In sedentary people, performing an intense bout of exercise and immediately replenishing spent calories doesn't affect next day metabolism." Being unable to reject that statement doesn't immediately confirm it in statistical terms, but even if that were the case, it is such a lousy statement to begin with, that any headlines coming out of it are outright misinformation. Both groups were equally sedentary and that might have nothing to do with any observed effects. [0]: https://en.wikipedia.org/wiki/Null_hypothesis https://en.wikipedia.org/wiki/Null_hypothesis
  - anjc 8y ago
    Ah thanks, I understand this aspect. As I say, they simply failed to rejected the null hypothesis, and their approach here seems perfectly valid. The authors did overreach in their interpretation of the result. But OP said ANOVA is wrong, and that the alpha value is wrong. I don't understand how they can say this without understanding this domain, the intervention, and knowing what effect size could be expected. Maybe α=0.05 is perfectly reasonable here.
ourmandave 8y ago
Makes you wonder what they're selling. Get a few sites to pick up a bogus scientific study article supporting a new product they're releasing is a thing.
- iheartpotatoes 8y ago
  Right? This feels like a foundational article used to build bullshit credentials somewhere else by planting bogus citations that no one will check because they are >1 level deep. Hell, people don't even check 1-level deep citations.

glastra 8y ago

The results themselves are not wrong per se. Reaching this conclusion from those results, however, is frankly insulting and should make all authors of the paper ashamed. > These data indicate that physical inactivity (e.g., sitting ~13.5 hrs/day and <4,000 steps/day) creates a condition whereby people become "resistant" to the metabolic improvements that are typically derived from an acute bout of aerobic exercise (i.e., exercise resistance). Also note that the researchers _immediately_ re-fed the group who participated in the intense exercise: > The additional energy expenditure from the exercise session was estimated via indirect calorimetry. This energy was then replaced with additional caloric intake with the post-exercise dinner (Clif® Builder’s®; ~40%/30%/30% of calories from carbohydrate, fat, and protein, respectively). This is especially bad because it isn't mentioned anywhere in the abstract (I checked the full-text). Shit science.

cjslep 8y ago

I also wanted to dig into this. The null hypothesis and burden-of-proofs change between p tests and equivalence testing, right? P tests: null hypothesis is that there is nothing different, burden of proof to suggest there is a difference. Equivalence testing: null hypothesis is that there is something different, burden of proof to suggest there is no difference. But maybe this is just a misapplied p-test? So while the "RESULTS" section of the paper is fine: No differences between trials ( P > 0.05) were found in the overall plasma triglyceride, glucose, or insulin responses during the HFGTT. The "CONCLUSION" section is where they invert the statistics of their own study and literally suggest a p-test-able difference despite their results section literally concluding a p-test-able non-difference: These data indicate that physical inactivity (e.g., sitting ~13.5 hrs/day and <4,000 steps/day) creates a condition whereby people become "resistant" to the metabolic improvements that are typically derived from an acute bout of aerobic exercise (i.e., exercise resistance).

Darmani 8y ago
Yes, they used a test designed to find a difference between two groups, and it failed to report a difference. Your analysis of that is correct, although the terminology "p-test" to describe "a statistical test designed to find a difference" is not. I don't know the general term for this class of tests. The definition of p-values is a bit wonky (there's actually a set of null hypotheses rather than a single one, and it involves a supremum over that set), but the line "if we're wrong, then the chance of seeing this data is at most p" is pretty accurate. All kinds of frequentist statistical tests use them, including equivalence tests.

photonios 8y ago

Excellent analysis! For those interested, here's the study the article quotes: https://www.ncbi.nlm.nih.gov/pubmed/30763169 https://www.ncbi.nlm.nih.gov/pubmed/30763169

anjc 8y ago

Can you explain this common mistake further? > They're claiming there is no difference between groups because they were unable to show that, if there was an inherent difference, the chance they'd see these results is at most 5%. It seems to me that they're claiming that they failed to reject the null hypothesis, the end. I.e. according to this statistic, there was no significant difference in the means between all groups in the ANOVA. This was a small sample size, but the analysis seems to include multiple trials for each participant. This seems like a standard use of ANOVA to me. So how can you say that the alpha value is wrong just based on this one sentence from the abstract? It seems to me that you'd have to know the domain, know what effect to expect, and so on

glastra 8y ago
Not OP, but [0]: > If the data do not contradict the null hypothesis, then only a weak conclusion can be made: namely, that the observed data set provides no strong evidence against the null hypothesis. In this case, because the null hypothesis could be true or false, in some contexts this is interpreted as meaning that the data give insufficient evidence to make any conclusion; in other contexts it is interpreted as meaning that there is no evidence to support changing from a currently useful regime to a different one. Also, keep in mind that in this study, the intervention (the difference between both groups) was, basically: perform an intense 1-hour bout of exercise and immediately replenish all calories spent. The null hypothesis is, thus: "In sedentary people, performing an intense bout of exercise and immediately replenishing spent calories doesn't affect next day metabolism." Being unable to reject that statement doesn't immediately confirm it in statistical terms, but even if that were the case, it is such a lousy statement to begin with, that any headlines coming out of it are outright misinformation. Both groups were equally sedentary and that might have nothing to do with any observed effects. [0]: https://en.wikipedia.org/wiki/Null_hypothesis https://en.wikipedia.org/wiki/Null_hypothesis
- anjc 8y ago
  Ah thanks, I understand this aspect. As I say, they simply failed to rejected the null hypothesis, and their approach here seems perfectly valid. The authors did overreach in their interpretation of the result. But OP said ANOVA is wrong, and that the alpha value is wrong. I don't understand how they can say this without understanding this domain, the intervention, and knowing what effect size could be expected. Maybe α=0.05 is perfectly reasonable here.

ourmandave 8y ago

Makes you wonder what they're selling. Get a few sites to pick up a bogus scientific study article supporting a new product they're releasing is a thing.

iheartpotatoes 8y ago
Right? This feels like a foundational article used to build bullshit credentials somewhere else by planting bogus citations that no one will check because they are >1 level deep. Hell, people don't even check 1-level deep citations.