8 ms·

None of your criticisms from wikipedia pertain to this publication. The paragraph is essentially making the argument that a randomized, double blind (DB), place

None of your criticisms from wikipedia pertain to this publication. The paragraph is essentially making the argument that a randomized, double blind (DB), placebo controlled (PC) clinical trial (RCT) is unlikely to show that curcurmin is effective. yet, the published article is exactly that, and shows a statistically measurable impact on a clinically meaningful endpoint: a direct refutation of the paragraph. the fact the "interference properties" and being an "unstable, reactive, non-bioavailable compound" is subsumed by the study design. claims to have studied a bioavailable form of curcumin This is not really relevant to the claim or conclusions. could it be that the bioavailability matters? maybe. could it be that curcurmin is a trick molecule to study? yes. but come what may, this is direct evidence (not proof) of a cause-effect relationship between this treatment and the outcome. the only thing this effects is the chance that other forms of curcurmin will be comparable (and it might have consequences for regulatory status by the FDA). The study also notes its sample was small This is a very weak criticism when the study is positive (i.e. shows an effect). when the sample size (and statistical power) of the study is small, you need a BIG EFFECT SIZE to be positive. The fact that the trial was positive actually means that it's more likely to be clinically important, because a big effect size means it protects memory in a dramatic way. This is like penicillin: in the pre-antibiotic era, how many cases of pneumonia do you need to treat in order to be satisfied that penicillin is effective? Not bloody many, because the effect size is so dramatic. Likewise for parachutes. few are the therapies that fall into this category, but when you have an inkling that one does following a DB, PC, RCT, you better pay attention. So does it need to be replicated? definitely. does the sponsorship matter? definitely; they could have cheated in some way. until then does it make sense to prescribe this to seniors, or encourage them to increase curcurmin in their diet? quite possibly, given the presumably low rate of adverse effects. Bottom line: this study is not the end-all be-all of curcurmin and memory, but there is legitimate early evidence that the emperor has clothes. edits: mainly spelling, formatting

closed 9y ago
When a small sample size produces a big effect, it does not mean you can be more confident in the effect! All things equal, you'd expect confidence intervals / credible regions to be very large. Another way to say this, is that since the measurement error is large, you expect spurious findings that result from something like p-hacking to be large (they have to be to be significant).
- roganp 9y ago
  This. This is a relevant podcast featuring John Ioannidis (author of "Why Most Published Research Findings Are False.") http://www.econtalk.org/archives/2018/01/john_ioannidis.html http://www.econtalk.org/archives/2018/01/john_ioannidis.html
- vanderZwan 9y ago
  What annoys me as a someone not trained in statistics is that these discussions are always abstract in-principle arguments about what proper statistics should be like. Meanwhile, we have a real paper with real p-values, why not discuss the validity of that? > SRT Consistent Long-Term Retrieval improved with curcumin (ES = 0.63, p = 0.002) but not with placebo (ES = 0.06, p = 0.8; between-group: ES = 0.68, p = 0.05). Curcumin also improved SRT Total (ES = 0.53, p = 0.002), visual memory (BVMT-R Recall: ES = 0.50, p = 0.01; BVMT-R Delay: ES = 0.51, p = 0.006), and attention (ES = 0.96, p < 0.0001) compared with placebo (ES = 0.28, p = 0.1; between-group: ES = 0.67, p = 0.04). FDDNP binding decreased significantly in the amygdala with curcumin (ES = −0.41, p = 0.04) compared with placebo (ES = 0.08, p = 0.6; between-group: ES = 0.48, p = 0.07). In the hypothalamus, FDDNP binding did not change with curcumin (ES = −0.30, p = 0.2), but increased with placebo (ES = 0.26, p = 0.05; between-group: ES = 0.55, p = 0.02). I can see that there are a lot of p-values around 0.05, but that the supposed improvements have much lower p-values: (ES = 0.63, p = 0.002) SRT Consistent Long-Term Retrieval (ES = 0.53, p = 0.002) SRT Total (ES = 0.50, p = 0.01) BVMT-R Recall (ES = 0.51, p = 0.006) BVMT-R Delay (ES = 0.96, p < 0.0001) attention What does this imply? Is it a good sign, or does it actually make it more fishy? pythonslange's comment[0] seems to suggest the latter, claiming that none of the other pre-registered tests are mentioned, without any explanation why. (eyeing that "attention" one, if this does hold up under scrutiny, could I expect tumeric-based ADD medication somewhere in the future, without all the nasty side-effects of my current amphetamine-based options?) [0] https://news.ycombinator.com/item?id=16229693 https://news.ycombinator.com/item?id=16229693
  - roganp 9y ago
    Here is the problem: If there is an actual large effect size, then you can detect this in small samples. However, the opposite is not true. In a small sample, spurious large effects are MORE likely, not less likely (outliers have a greater impact in a small sample). See: http://andrewgelman.com/2017/08/16/also-holding-back-progress-make-mistakes-label-correct-arguments-nonsensical/ http://andrewgelman.com/2017/08/16/also-holding-back-progres...
YeGoblynQueenne 9y ago
>> None of your criticisms from wikipedia pertain to this publication. The paragraph is essentially making the argument that a randomized, double blind (DB), placebo controlled (PC) clinical trial (RCT) is unlikely to show that curcurmin is effective. yet, the published article is exactly that, and shows a statistically measurable impact on a clinically meaningful endpoint: a direct refutation of the paragraph. I think, if there are solid reasons to believe that studies on curcumin will not turn up any useful results and then a study finds results it calls significant, the chances are that either that study is not designed very well, or its results are not significant, despite the claims of the authors. Apologies in advance for the controversial example, but this reminds me of the controversy about Daryl Bem's parapsychology research a few years ago- the author claimed his study was methodologically unassailable, but of course he was making an absurd claim (essentially, that people can receive information from the future via ESP). Edit: removed cruft.
- coldtea 9y ago
  >I think, if there are solid reasons to believe that studies on curcumin will not turn up any useful results and then a study finds results it calls significant, the chances are that either that study is not designed very well, or its results are not significant, despite the claims of the authors. If there are indeed "strong reasons" then the rest is a tautology. If there are not, then making such a conclusion is bias.

closed 9y ago

When a small sample size produces a big effect, it does not mean you can be more confident in the effect! All things equal, you'd expect confidence intervals / credible regions to be very large. Another way to say this, is that since the measurement error is large, you expect spurious findings that result from something like p-hacking to be large (they have to be to be significant).

roganp 9y ago
This. This is a relevant podcast featuring John Ioannidis (author of "Why Most Published Research Findings Are False.") http://www.econtalk.org/archives/2018/01/john_ioannidis.html http://www.econtalk.org/archives/2018/01/john_ioannidis.html
vanderZwan 9y ago
What annoys me as a someone not trained in statistics is that these discussions are always abstract in-principle arguments about what proper statistics should be like. Meanwhile, we have a real paper with real p-values, why not discuss the validity of that? > SRT Consistent Long-Term Retrieval improved with curcumin (ES = 0.63, p = 0.002) but not with placebo (ES = 0.06, p = 0.8; between-group: ES = 0.68, p = 0.05). Curcumin also improved SRT Total (ES = 0.53, p = 0.002), visual memory (BVMT-R Recall: ES = 0.50, p = 0.01; BVMT-R Delay: ES = 0.51, p = 0.006), and attention (ES = 0.96, p < 0.0001) compared with placebo (ES = 0.28, p = 0.1; between-group: ES = 0.67, p = 0.04). FDDNP binding decreased significantly in the amygdala with curcumin (ES = −0.41, p = 0.04) compared with placebo (ES = 0.08, p = 0.6; between-group: ES = 0.48, p = 0.07). In the hypothalamus, FDDNP binding did not change with curcumin (ES = −0.30, p = 0.2), but increased with placebo (ES = 0.26, p = 0.05; between-group: ES = 0.55, p = 0.02). I can see that there are a lot of p-values around 0.05, but that the supposed improvements have much lower p-values: (ES = 0.63, p = 0.002) SRT Consistent Long-Term Retrieval (ES = 0.53, p = 0.002) SRT Total (ES = 0.50, p = 0.01) BVMT-R Recall (ES = 0.51, p = 0.006) BVMT-R Delay (ES = 0.96, p < 0.0001) attention What does this imply? Is it a good sign, or does it actually make it more fishy? pythonslange's comment[0] seems to suggest the latter, claiming that none of the other pre-registered tests are mentioned, without any explanation why. (eyeing that "attention" one, if this does hold up under scrutiny, could I expect tumeric-based ADD medication somewhere in the future, without all the nasty side-effects of my current amphetamine-based options?) [0] https://news.ycombinator.com/item?id=16229693 https://news.ycombinator.com/item?id=16229693
- roganp 9y ago
  Here is the problem: If there is an actual large effect size, then you can detect this in small samples. However, the opposite is not true. In a small sample, spurious large effects are MORE likely, not less likely (outliers have a greater impact in a small sample). See: http://andrewgelman.com/2017/08/16/also-holding-back-progress-make-mistakes-label-correct-arguments-nonsensical/ http://andrewgelman.com/2017/08/16/also-holding-back-progres...

YeGoblynQueenne 9y ago

>> None of your criticisms from wikipedia pertain to this publication. The paragraph is essentially making the argument that a randomized, double blind (DB), placebo controlled (PC) clinical trial (RCT) is unlikely to show that curcurmin is effective. yet, the published article is exactly that, and shows a statistically measurable impact on a clinically meaningful endpoint: a direct refutation of the paragraph. I think, if there are solid reasons to believe that studies on curcumin will not turn up any useful results and then a study finds results it calls significant, the chances are that either that study is not designed very well, or its results are not significant, despite the claims of the authors. Apologies in advance for the controversial example, but this reminds me of the controversy about Daryl Bem's parapsychology research a few years ago- the author claimed his study was methodologically unassailable, but of course he was making an absurd claim (essentially, that people can receive information from the future via ESP). Edit: removed cruft.

coldtea 9y ago
>I think, if there are solid reasons to believe that studies on curcumin will not turn up any useful results and then a study finds results it calls significant, the chances are that either that study is not designed very well, or its results are not significant, despite the claims of the authors. If there are indeed "strong reasons" then the rest is a tautology. If there are not, then making such a conclusion is bias.