6 ms·

>> Small study >Almost entirely irrelevant, provided it's properly powered--and based on the effect size and p-values, it is. Sorry to state this so strongly

>> Small study >Almost entirely irrelevant, provided it's properly powered--and based on the effect size and p-values, it is. Sorry to state this so strongly but I feel it's important to point out that this is misinformation of the most egregious kind. There is no examination of statistical power in this article whatsoever so there is no basis for your claim that this study is properly powered. Large effect sizes are not evidence of adequate statistical power. Quite the opposite is true. E.g. See this article: https://www.nature.com/articles/nrn3475/figures/5 https://www.nature.com/articles/nrn3475/figures/5 'Effect inflation is worst for small, low-powered studies, which can only detect effects that happen to be large.'

sideshowb 9y ago
Am I right in thinking that statistical power is the true positive rate, so a test that always rejected the null hypothesis (whether correctly or incorrectly) would have the highest possible statistical power?
- closed 9y ago
  No, statistical power as a concept is independent of the observed result of a test. It is widely misunderstood though, so often people try to use an observed result to justify low power. http://andrewgelman.com/2017/08/17/just-google-despite-limited-statistical-power/ http://andrewgelman.com/2017/08/17/just-google-despite-limit...
  - sideshowb 9y ago
    Is the wikipedia page on statistical power plain wrong then? "The power of a binary hypothesis test is the probability that the test correctly rejects the null hypothesis (H0) when a specific alternative hypothesis (H1) is true. The statistical power ranges from 0 to 1, and as statistical power increases, the probability of making a type 2 error decreases. For a type 2 error probability of β, the corresponding statistical power is 1-β. " So according to that, if I always reject H0 whether rightly or wrongly, that means β=0 and power=1.
dumbneurologist 9y ago
Sorry to state this so strongly but I feel it's important to point out that this is misinformation of the most egregious kind. There is no examination of statistical power in this article whatsoever so there is no basis for your claim that this study is properly powered. The power is only relevant when you fail to detect an effect. This is a positive study, so the power is simply not relevant to criticizing that particular study. The nature article you reference talks about literature in aggregate, and the problem created by small studies in general, but it does not allow you to say anything about a single positive study. In fact, the place where those very authors draw their primary conclusion, they state: Inflated effect estimates make it difficult to determine an adequate sample size for replication studies, increasing the probability of type II errors Which means: when you try to replicate it, you might fail because you thought the real effect size was bigger than it is, based on your estimated effect size with small n. That's well and good, but is a problem for the replicating authors, not for these. An even more direct point related to the current article: it was preregistered, so you know the authors have not been farming well-designed but small curcurmin studies until they did one than that was positive by chance, and then published it. The act of pre-registration goes a very long way toward addressing the issues raised by Button et al.
- closed 9y ago
  > The power is only relevant when you fail to detect an effect. This is simply not true. It is sometimes called the fallacy of "what does not kill my statistical significance makes it stronger”. See http://andrewgelman.com/2017/08/17/just-google-despite-limited-statistical-power/ http://andrewgelman.com/2017/08/17/just-google-despite-limit...
- martingoodson 9y ago
  The power is only relevant when you fail to detect an effect. This is absolutely false. Small, low-powered studies are more likely to produce false results than larger studies. This was well-documented by John Ioannidis over ten years ago, in work that should be required reading for all publishing scientists: http://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124 http://journals.plos.org/plosmedicine/article?id=10.1371/jou...

sideshowb 9y ago

Am I right in thinking that statistical power is the true positive rate, so a test that always rejected the null hypothesis (whether correctly or incorrectly) would have the highest possible statistical power?

closed 9y ago
No, statistical power as a concept is independent of the observed result of a test. It is widely misunderstood though, so often people try to use an observed result to justify low power. http://andrewgelman.com/2017/08/17/just-google-despite-limited-statistical-power/ http://andrewgelman.com/2017/08/17/just-google-despite-limit...
- sideshowb 9y ago
  Is the wikipedia page on statistical power plain wrong then? "The power of a binary hypothesis test is the probability that the test correctly rejects the null hypothesis (H0) when a specific alternative hypothesis (H1) is true. The statistical power ranges from 0 to 1, and as statistical power increases, the probability of making a type 2 error decreases. For a type 2 error probability of β, the corresponding statistical power is 1-β. " So according to that, if I always reject H0 whether rightly or wrongly, that means β=0 and power=1.

dumbneurologist 9y ago

Sorry to state this so strongly but I feel it's important to point out that this is misinformation of the most egregious kind. There is no examination of statistical power in this article whatsoever so there is no basis for your claim that this study is properly powered. The power is only relevant when you fail to detect an effect. This is a positive study, so the power is simply not relevant to criticizing that particular study. The nature article you reference talks about literature in aggregate, and the problem created by small studies in general, but it does not allow you to say anything about a single positive study. In fact, the place where those very authors draw their primary conclusion, they state: Inflated effect estimates make it difficult to determine an adequate sample size for replication studies, increasing the probability of type II errors Which means: when you try to replicate it, you might fail because you thought the real effect size was bigger than it is, based on your estimated effect size with small n. That's well and good, but is a problem for the replicating authors, not for these. An even more direct point related to the current article: it was preregistered, so you know the authors have not been farming well-designed but small curcurmin studies until they did one than that was positive by chance, and then published it. The act of pre-registration goes a very long way toward addressing the issues raised by Button et al.

closed 9y ago
> The power is only relevant when you fail to detect an effect. This is simply not true. It is sometimes called the fallacy of "what does not kill my statistical significance makes it stronger”. See http://andrewgelman.com/2017/08/17/just-google-despite-limited-statistical-power/ http://andrewgelman.com/2017/08/17/just-google-despite-limit...
martingoodson 9y ago
The power is only relevant when you fail to detect an effect. This is absolutely false. Small, low-powered studies are more likely to produce false results than larger studies. This was well-documented by John Ioannidis over ten years ago, in work that should be required reading for all publishing scientists: http://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124 http://journals.plos.org/plosmedicine/article?id=10.1371/jou...