3 ms·

Hi, actual OP here. I said that failing to show a statistically significant difference is not the same as showing equivalence with statistical significance. Te

Hi, actual OP here. I said that failing to show a statistically significant difference is not the same as showing equivalence with statistical significance. Tests designed for the former cannot do the latter with any amount of data. You need a different test for that, or at least the same underlying statistical test used in a different manner. Can you explain what about this implies that "ANOVA and the alpha value are wrong?" I'll confess to having never really learned ANOVA, but it sounds like it's a family of models generalizing the t-test. You can indeed perform equivalence and non-inferiority testing with the t-test, as I did in my OOPSLA 2018 paper. You just have to use it in a different way than it sounds like they did here.

anjc 8y ago
Thanks for reply. Regarding alpha value: you seem to say that with a small sample size, it'd be possible to see no effect between groups. I think you were implying that they should've been testing p<0.0001 or something? But I'm not sure how to choose this value, without knowing the difference between means that you expect for the medical intervention.
- Darmani 8y ago
  So, as I understand it: how do you pick a P such that, if you fail to prove there is a difference with p<P, then there is no difference? If you compute the power of your test, you can do this, so long as you know the expected difference of your means. Like you implied, that is not realistic. Do you do something else entirely: https://en.m.wikipedia.org/wiki/Equivalence_test https://en.m.wikipedia.org/wiki/Equivalence_test