5 ms·

Can you explain this common mistake further? > They're claiming there is no difference between groups because they were unable to show that, if there was an in

by anjc 8y ago

Can you explain this common mistake further? > They're claiming there is no difference between groups because they were unable to show that, if there was an inherent difference, the chance they'd see these results is at most 5%. It seems to me that they're claiming that they failed to reject the null hypothesis, the end. I.e. according to this statistic, there was no significant difference in the means between all groups in the ANOVA. This was a small sample size, but the analysis seems to include multiple trials for each participant. This seems like a standard use of ANOVA to me. So how can you say that the alpha value is wrong just based on this one sentence from the abstract? It seems to me that you'd have to know the domain, know what effect to expect, and so on

glastra 8y ago
Not OP, but [0]: > If the data do not contradict the null hypothesis, then only a weak conclusion can be made: namely, that the observed data set provides no strong evidence against the null hypothesis. In this case, because the null hypothesis could be true or false, in some contexts this is interpreted as meaning that the data give insufficient evidence to make any conclusion; in other contexts it is interpreted as meaning that there is no evidence to support changing from a currently useful regime to a different one. Also, keep in mind that in this study, the intervention (the difference between both groups) was, basically: perform an intense 1-hour bout of exercise and immediately replenish all calories spent. The null hypothesis is, thus: "In sedentary people, performing an intense bout of exercise and immediately replenishing spent calories doesn't affect next day metabolism." Being unable to reject that statement doesn't immediately confirm it in statistical terms, but even if that were the case, it is such a lousy statement to begin with, that any headlines coming out of it are outright misinformation. Both groups were equally sedentary and that might have nothing to do with any observed effects. [0]: https://en.wikipedia.org/wiki/Null_hypothesis https://en.wikipedia.org/wiki/Null_hypothesis
- anjc 8y ago
  Ah thanks, I understand this aspect. As I say, they simply failed to rejected the null hypothesis, and their approach here seems perfectly valid. The authors did overreach in their interpretation of the result. But OP said ANOVA is wrong, and that the alpha value is wrong. I don't understand how they can say this without understanding this domain, the intervention, and knowing what effect size could be expected. Maybe α=0.05 is perfectly reasonable here.
  - Darmani 8y ago
    Hi, actual OP here. I said that failing to show a statistically significant difference is not the same as showing equivalence with statistical significance. Tests designed for the former cannot do the latter with any amount of data. You need a different test for that, or at least the same underlying statistical test used in a different manner. Can you explain what about this implies that "ANOVA and the alpha value are wrong?" I'll confess to having never really learned ANOVA, but it sounds like it's a family of models generalizing the t-test. You can indeed perform equivalence and non-inferiority testing with the t-test, as I did in my OOPSLA 2018 paper. You just have to use it in a different way than it sounds like they did here.
    - anjc 8y ago
      Thanks for reply. Regarding alpha value: you seem to say that with a small sample size, it'd be possible to see no effect between groups. I think you were implying that they should've been testing p<0.0001 or something? But I'm not sure how to choose this value, without knowing the difference between means that you expect for the medical intervention.
      - Darmani 8y ago
        So, as I understand it: how do you pick a P such that, if you fail to prove there is a difference with p<P, then there is no difference? If you compute the power of your test, you can do this, so long as you know the expected difference of your means. Like you implied, that is not realistic. Do you do something else entirely: https://en.m.wikipedia.org/wiki/Equivalence_test https://en.m.wikipedia.org/wiki/Equivalence_test

by anjc 8y ago

glastra 8y ago

Not OP, but [0]: > If the data do not contradict the null hypothesis, then only a weak conclusion can be made: namely, that the observed data set provides no strong evidence against the null hypothesis. In this case, because the null hypothesis could be true or false, in some contexts this is interpreted as meaning that the data give insufficient evidence to make any conclusion; in other contexts it is interpreted as meaning that there is no evidence to support changing from a currently useful regime to a different one. Also, keep in mind that in this study, the intervention (the difference between both groups) was, basically: perform an intense 1-hour bout of exercise and immediately replenish all calories spent. The null hypothesis is, thus: "In sedentary people, performing an intense bout of exercise and immediately replenishing spent calories doesn't affect next day metabolism." Being unable to reject that statement doesn't immediately confirm it in statistical terms, but even if that were the case, it is such a lousy statement to begin with, that any headlines coming out of it are outright misinformation. Both groups were equally sedentary and that might have nothing to do with any observed effects. [0]: https://en.wikipedia.org/wiki/Null_hypothesis https://en.wikipedia.org/wiki/Null_hypothesis

anjc 8y ago
Ah thanks, I understand this aspect. As I say, they simply failed to rejected the null hypothesis, and their approach here seems perfectly valid. The authors did overreach in their interpretation of the result. But OP said ANOVA is wrong, and that the alpha value is wrong. I don't understand how they can say this without understanding this domain, the intervention, and knowing what effect size could be expected. Maybe α=0.05 is perfectly reasonable here.
- Darmani 8y ago
  Hi, actual OP here. I said that failing to show a statistically significant difference is not the same as showing equivalence with statistical significance. Tests designed for the former cannot do the latter with any amount of data. You need a different test for that, or at least the same underlying statistical test used in a different manner. Can you explain what about this implies that "ANOVA and the alpha value are wrong?" I'll confess to having never really learned ANOVA, but it sounds like it's a family of models generalizing the t-test. You can indeed perform equivalence and non-inferiority testing with the t-test, as I did in my OOPSLA 2018 paper. You just have to use it in a different way than it sounds like they did here.
  - anjc 8y ago
    Thanks for reply. Regarding alpha value: you seem to say that with a small sample size, it'd be possible to see no effect between groups. I think you were implying that they should've been testing p<0.0001 or something? But I'm not sure how to choose this value, without knowing the difference between means that you expect for the medical intervention.
    - Darmani 8y ago
      So, as I understand it: how do you pick a P such that, if you fail to prove there is a difference with p<P, then there is no difference? If you compute the power of your test, you can do this, so long as you know the expected difference of your means. Like you implied, that is not realistic. Do you do something else entirely: https://en.m.wikipedia.org/wiki/Equivalence_test https://en.m.wikipedia.org/wiki/Equivalence_test