9 ms·

Small study, probably not preregistered (didn't find anything about it in the description). Likely Publication Bias or p-hacking. I'm starting to get interested

by hannob 9y ago

Small study, probably not preregistered (didn't find anything about it in the description). Likely Publication Bias or p-hacking. I'm starting to get interested when it's been independently replicated.

loeg 9y ago
Results too good to be true, as well :-).
soneca 9y ago
Great "credibility check". There should be a "nutriction facts table" for scientific papers quoted at articles. Sample size, preregistration, replication, p-value. At least a practice to include that in the abstract
- pjmorris 9y ago
  Maybe along the lines of 'A statistical definition for reproducibility and replicability'? I think there are more attributes to be included, but it seems like a good place to start on a 'nutrition label' for papers. [0] https://www.biorxiv.org/content/early/2016/07/29/066803 https://www.biorxiv.org/content/early/2016/07/29/066803 [1] https://cran.r-project.org/web/packages/scifigure/index.html https://cran.r-project.org/web/packages/scifigure/index.html
platz 9y ago
What is the power & effect size?
- bildung 9y ago
  Effect size is the amount of change that has been observed, in this case the ability to memorize things. Significance is the probability of the result being coincidence. So "highly significant" only means "most probably not coincidence". That means you can have a "highly significant" result where the actual result (= effect size) is tiny. In study design, both are related: The higher the number of test persons, the easier it is to archive significance. Therefore, proper design would be to first guesstimate (based on existing literature etc.) the effect size, then define the minimum number of test persons necessary.
deleted 9y ago
[deleted]
piyh 9y ago
If every study on tumeric, tea, chocolate, and coffee was fully accurate we'd all be immortal by now.
- ShabbosGoy 9y ago
  Aren’t we? People are living longer than ever before. The issue is in determining effect size. I’m sure some amount of all those things you mentioned is “good” in some sense, but in what quantity and for how long?
  - azemetre 9y ago
    People are "living longer" because child mortality has fallen dramatically.
    - deleted 9y ago
      [deleted]
    - croon 9y ago
      If in making those statistics one didn't account for that I'd be very disappointed, but it doesn't look like that's the case. [1][2] [1] http://life-span.healthgrove.com/l/51/50 http://life-span.healthgrove.com/l/51/50 [2] https://www.ncbi.nlm.nih.gov/books/NBK62587/ https://www.ncbi.nlm.nih.gov/books/NBK62587/
    - klmr 9y ago
      Not only. Life expectancy has increased in every age group percentile. But the effect is obviously not due to tea, coffee and turmeric. Rather, it’s due, mainly, to (a) better general nutrition, (b) better hygiene, and (c) modern medicine, especially antibiotics and vaccines.
- Tepix 9y ago
  I'm doing my part!
- vanderZwan 9y ago
  The problem you describe is a real one, but a bit different than the other criticism raised here so far. Chocolate has the Mars Company and other Big Sugar companies behind it pushing for justifying chocolate as a "health food" in the public mind. It is very obvious that they are wiling to distort the truth for better sales; people who can ease their conscious about eating unhealthy sweets buy more chocolate[0]. There might be some real effect to eating raw cocoa, but that does not really translate to your average chocolate bar with insane amounts of fat and sugar. Tea/Coffee is kind of similar, as would be the "drink x glasses of water per day"-advice, which is a distortion of science by bottled water companies (fruits, vegetables and other food contains a ton of water already). For this reason I also don't buy that salt is safe: there are very strong vested interests in the food industry that want salt to be (considered) safe to sell more food, whereas I cannot see a comparable bias on the "eat less salt"-side. For this research the situation seems to be have some of the mentioned problems, but not to the same degree: On the one hand, that tumeric has an in-vitro effect is pretty well established. The issue is that the normal form we put on our food is probably not absorbed in a dosage that has a significant effect. However, this was a trial with a bio-available form, in a high enough dosage, so that obvious flaw is out of the picture. On the other hand, that bio-available form is a product, so the company behind it wants it to be effective. Still, Big Pharma is under more scrutiny than Big Sugar, so I have cautious hopes this might actually work out in the end (and it won't be sold in the form of calorie-rich sweets at least). Admittedly, I'm biased here: my grandmother on my mother's side, and my grandfather on my father's side both had Alzheimers (or something very similar to it at least). My parents are now their late sixties, I would really like to see them stay healthy and happy for a few more decades. [0] https://www.vox.com/science-and-health/2017/10/18/15995478/chocolate-health-benefits-heart-disease https://www.vox.com/science-and-health/2017/10/18/15995478/c...
sov 9y ago
> Small study Almost entirely irrelevant, provided it's properly powered--and based on the effect size and p-values, it is. Furthermore, for an 18 month trial, 40 people is larger than most. > probably not preregistered I mean, true, and we should definitely promote pre-reg, but, 1. so few are preregistered that I'm not sure it's prudent to disregard a study based on this and 2. it's an academic study, neither a clinical trial nor a public-health decree--not a typical study type that necessitates pre-registration. > Likely Publication Bias or p-hacking. What? You're arrived at this conclusion based on... the fact that it's "small" and you couldn't find preregistration about it? > I'm starting to get interested when it's been independently replicated. Yeah, if only there were another trial that showed a positive cognitive effect of bioavailable curcumin... https://www.ncbi.nlm.nih.gov/pubmed/25277322 https://www.ncbi.nlm.nih.gov/pubmed/25277322 I mean, the title of this post is by far the most wrong thing about this article, as... 1. it's not turmeric, it's curcumin, which is about 2% of turmeric by mass (so, you'd need to eat ~4.5g turmeric daily to match study doses). 2. curcumin doesn't even have any reasonable bioavilability by itself, so they're using a bioavailable analogue.
- andrewjl 9y ago
  What prevents researchers from running studies like these on the order of thousands of individuals, as opposed to 40 or 100?
  - shanedrgn 9y ago
    Funding
  - sov 9y ago
    A lot of things that basically boil down to budget, optimization and space available. If you have a drug that you believe you can show a strong effect with the effect group at ~15 people, maybe you run the trial with 40 (15 e.g., 15 control, plus wiggle room for drop-outs over the 18-month course and extra certainty). Check the last few paragraphs--"The authors thank Shayna Greenberg, Dev Darshan Khalsa, and Anya Rosensteel for help in recruitment, data management, and study coordination; John Williams, University of California, Los Angeles, Nuclear Medicine Clinic, for performing the PET scans; and Vladimir Kepe, Cleveland Clinic, for developing the FDDNP-PET analysis procedures for proteinopathies in patients with neurodegenerative diseases." Add all that to the effort involved in the actual chemical extraction of the bioavailable curcumin analogue and it just doesn't make sense to add any more people than you need.
piracy1 9y ago
It's possible, but I think (not that I'm an expert) that this is not the first piece of evidence to suggest this. A year or two was recently speaking with a researcher who studies Alzheimer's and he had told me about how in cultures where more curry and turmeric was consumed there was significantly less Alzherimer's. I'll try to find a source for this and update my comment.
- lstyls 9y ago
  Not only is that anecdotal, but even if it's true it is correlative in the weakest sense. There are undoubtedly countless other features for which "cultures" (how does one define this concretely btw) with low Alzheimer's rates also are outliers. Here is a great demonstration of how correlation does not imply causation: http://www.tylervigen.com/spurious-correlations http://www.tylervigen.com/spurious-correlations.
  - piracy1 9y ago
    Sure. But I'm saying that bit of information would become interesting given the original post which supports a similar idea. I'm not making an assertion of fact simply bringing to mind another data point. Also here. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2781139/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2781139/ It would seem that this is a rather well documented and often researched thing. Not just a one-off p-hack. I believe the research I was thinking of was specifically this part of the overview above.: Various studies and research[9,10] results indicate a lower incidence and prevalence of AD in India. The prevalence of AD among adults aged 70-79 years in India is 4.4 times less than that of adults aged 70-79 years in the United States.[9] Researchers investigated the association between the curry consumption and cognitive level in 1010 Asians between 60 and 93 years of age. The study found that those who occasionally ate curry (less than once a month) and often (more than once a month) performed better on a standard test (MMSE) of cognitive function than those who ate curry never or rarely.[10] Sources: 9. Pandav R, Belle SH, DeKosky ST. Apolipoprotein E polymorphism and Alzheimer's disease: The Indo-US cross-national dementia study. Arch Neurol. 2000;57:824–30. [PubMed] 10. Ng TP, Chiam PC, Lee T, Chua HC, Lim L, Kua EH. Curry consumption and cognitive function in the elderly. Am J Epidemiol. 2006;164:898–906. [PubMed]
EnderWT 9y ago
Examine.com has a page for curcumin. They cite 5 sources with notable impacts on reducing depression, but they summarize with this: "Curcumin seems to be somewhat more effective than placebo in reducing symptoms of depression. It may take 2-3 months to see any outcomes. Skepticism is warranted though, as the studies comparing curcumin to placebo were not well designed and produced effect sizes not too far apart, even though the differences were statistically significant." https://examine.com/supplements/curcumin/#hem-depression https://examine.com/supplements/curcumin/#hem-depression
roystonvassey 9y ago
This is becoming an increasingly worrying trend - to bunk any analysis with a comment on 'small sample size, biases' that somehow moves to the top of the comment pile. Statistically, there is no such thing as a 'small sample size', in isolation. It is always related to the power and effect that the researchers want to work with. Moreover, even if a study is statistically insignificant, it DOES NOT mean that the opposite conclusion is true. It might often merit some further examination of cultural and historical trends. In this case, for instance, turmeric has been used for medicinal properties in Asian cultures for centuries. Theories of evolution suggest that this wouldn't be the case (and it would've died a natural death), if there wasn't some merit.
- carlmr 9y ago
  >In this case, for instance, turmeric has been used for medicinal properties in Asian cultures for centuries. Theories of evolution suggest that this wouldn't be the case (and it would've died a natural death), if there wasn't some merit. Lots of things that are completely useless have been used for thousands of years for their medicinal purposes. The evolution argument only holds if there were some negative effects here. Anything that doesn't do harm to the body can easily integrate into the huge body of knowledge called superstition. Eating it is tasty and doesn't hurt, there's nothing more needed to explain that Asian cultures have it as a medicinal herb. If it for example was good for you but caused somewhat unpleasant side-effects, your evolutionary argument might have more merit. Anything that works as well as homeopathy (it doesn't), but doesn't harm either (like homeopathy), won't be sorted out.
- afromatic 9y ago
  Moreover, even if a property is evolutionarily insignificant, it DOES NOT mean that it will be selected against.
- Retric 9y ago
  Statistics makes basic assumptions that are false. For example, most things don't fit a bell curve. These assumptions are more relevant in small sample sizes. Making them inherently unreliable. What's missing is not a p-value, but an estimate for the uncertainty around that p-value.
pytonslange 9y ago
It was registered as a clinical trial (https://clinicaltrials.gov/ct2/show/NCT01383161?term=NCT01383161&rank=1 https://clinicaltrials.gov/ct2/show/NCT01383161?term=NCT0138...). If you are concerned about p-hacking and replicability, what should worry you is that the outcome measures posted during registration are rather vague, i.e. ".. show less evidence of cognitive decline (as measured with neuropsychological assessments)" At baseline, they performed an extensive neuropsychologist test battery, including a bunch of subtests: -Trail Making Test A -WAIS-III Digit Symbol Substitution -WAIS-III Block Design Test -Rey-Osterrieth Complex Figure Test (copy) -Trail Making Test B -Stroop Interference -F.A.S. -Buschke-Fuld Selective Reminding Test [SRT] -Wechsler Memory Scale-3rd Edition [WMS-III] -Verbal Paired Associations I -Benton Visual Retention Test -Buschke-Fuld SRT -Rey-Osterrieth Complex Figure Test [recall] -WMS-III Verbal Paired Associations II -Boston Naming Test -Animal Naming Test The outcome measures they report are: -TMT-A -SRT -BVMT-R I would be _very_ surprised if those were the only tests they performed at followup after a 18 month (expensive) clinical trial. If these measures had been specified as the only outcome measures of interest, noone would question the results. But, when it was not registered, and when no other measures are reported, not even in the supplementary, the reader is left to wonder why that is.
- closed 9y ago
  That's a good point. After having seen much of the brain training literature (especially its early studies), this wouldn't surprise me at all.
- hanklazard 9y ago
  Excellent point. Outcome measures should have been more clearly defined in pre-registration. As it was done, p-hacking and selective reporting were still possible.
dumbneurologist 9y ago
agree that replication is important and that it's too early to believe fully, but when a study size is small and you still find a difference, then it implies the effect size is large. If you meet statistical significance, then diminishing n means increasing clinical significance.
- klmr 9y ago
  > but when a study size is small and you still find a difference, then it implies the effect size is large That’s only true if it holds across multiple studies. Otherwise it might simply be random variability. By your argument, a study with n=4 that showed an effect would imply a huge effect size. No. It implies that the measured signal is highly variable, and that you got lucky on 1:3 odds. Check out regression towards the mean [1]. (Note that if you’ve got a good underlying model of the variability, n=4 can work. In fact, it’s routinely done in specific applications, such as transcriptomics. But even there it’s far from ideal, and it’s supported by a stringent error model.) [1] https://en.wikipedia.org/wiki/Regression_toward_the_mean https://en.wikipedia.org/wiki/Regression_toward_the_mean
- pbhjpbhj 9y ago
  I flipped 5 coins, 4 came up heads, so heads must be >>80% likelihood??
dumbneurologist 9y ago
Except that it was preregistered, as described in the article. https://clinicaltrials.gov/ct2/show/NCT01383161?cond=NCT01383161&rank=1 https://clinicaltrials.gov/ct2/show/NCT01383161?cond=NCT0138... It's great to be skeptical, but it's really lame to be skeptical and lazy. It's a terrible combination that causes patients not to trust their doctors.

by hannob 9y ago

loeg 9y ago

Results too good to be true, as well :-).

soneca 9y ago

Great "credibility check". There should be a "nutriction facts table" for scientific papers quoted at articles. Sample size, preregistration, replication, p-value. At least a practice to include that in the abstract

pjmorris 9y ago
Maybe along the lines of 'A statistical definition for reproducibility and replicability'? I think there are more attributes to be included, but it seems like a good place to start on a 'nutrition label' for papers. [0] https://www.biorxiv.org/content/early/2016/07/29/066803 https://www.biorxiv.org/content/early/2016/07/29/066803 [1] https://cran.r-project.org/web/packages/scifigure/index.html https://cran.r-project.org/web/packages/scifigure/index.html

platz 9y ago

What is the power & effect size?

bildung 9y ago
Effect size is the amount of change that has been observed, in this case the ability to memorize things. Significance is the probability of the result being coincidence. So "highly significant" only means "most probably not coincidence". That means you can have a "highly significant" result where the actual result (= effect size) is tiny. In study design, both are related: The higher the number of test persons, the easier it is to archive significance. Therefore, proper design would be to first guesstimate (based on existing literature etc.) the effect size, then define the minimum number of test persons necessary.

deleted 9y ago

[deleted]

piyh 9y ago

If every study on tumeric, tea, chocolate, and coffee was fully accurate we'd all be immortal by now.

ShabbosGoy 9y ago
Aren’t we? People are living longer than ever before. The issue is in determining effect size. I’m sure some amount of all those things you mentioned is “good” in some sense, but in what quantity and for how long?
- azemetre 9y ago
  People are "living longer" because child mortality has fallen dramatically.
  - deleted 9y ago
    [deleted]
  - croon 9y ago
    If in making those statistics one didn't account for that I'd be very disappointed, but it doesn't look like that's the case. [1][2] [1] http://life-span.healthgrove.com/l/51/50 http://life-span.healthgrove.com/l/51/50 [2] https://www.ncbi.nlm.nih.gov/books/NBK62587/ https://www.ncbi.nlm.nih.gov/books/NBK62587/
  - klmr 9y ago
    Not only. Life expectancy has increased in every age group percentile. But the effect is obviously not due to tea, coffee and turmeric. Rather, it’s due, mainly, to (a) better general nutrition, (b) better hygiene, and (c) modern medicine, especially antibiotics and vaccines.
Tepix 9y ago
I'm doing my part!
vanderZwan 9y ago
The problem you describe is a real one, but a bit different than the other criticism raised here so far. Chocolate has the Mars Company and other Big Sugar companies behind it pushing for justifying chocolate as a "health food" in the public mind. It is very obvious that they are wiling to distort the truth for better sales; people who can ease their conscious about eating unhealthy sweets buy more chocolate[0]. There might be some real effect to eating raw cocoa, but that does not really translate to your average chocolate bar with insane amounts of fat and sugar. Tea/Coffee is kind of similar, as would be the "drink x glasses of water per day"-advice, which is a distortion of science by bottled water companies (fruits, vegetables and other food contains a ton of water already). For this reason I also don't buy that salt is safe: there are very strong vested interests in the food industry that want salt to be (considered) safe to sell more food, whereas I cannot see a comparable bias on the "eat less salt"-side. For this research the situation seems to be have some of the mentioned problems, but not to the same degree: On the one hand, that tumeric has an in-vitro effect is pretty well established. The issue is that the normal form we put on our food is probably not absorbed in a dosage that has a significant effect. However, this was a trial with a bio-available form, in a high enough dosage, so that obvious flaw is out of the picture. On the other hand, that bio-available form is a product, so the company behind it wants it to be effective. Still, Big Pharma is under more scrutiny than Big Sugar, so I have cautious hopes this might actually work out in the end (and it won't be sold in the form of calorie-rich sweets at least). Admittedly, I'm biased here: my grandmother on my mother's side, and my grandfather on my father's side both had Alzheimers (or something very similar to it at least). My parents are now their late sixties, I would really like to see them stay healthy and happy for a few more decades. [0] https://www.vox.com/science-and-health/2017/10/18/15995478/chocolate-health-benefits-heart-disease https://www.vox.com/science-and-health/2017/10/18/15995478/c...

sov 9y ago

> Small study Almost entirely irrelevant, provided it's properly powered--and based on the effect size and p-values, it is. Furthermore, for an 18 month trial, 40 people is larger than most. > probably not preregistered I mean, true, and we should definitely promote pre-reg, but, 1. so few are preregistered that I'm not sure it's prudent to disregard a study based on this and 2. it's an academic study, neither a clinical trial nor a public-health decree--not a typical study type that necessitates pre-registration. > Likely Publication Bias or p-hacking. What? You're arrived at this conclusion based on... the fact that it's "small" and you couldn't find preregistration about it? > I'm starting to get interested when it's been independently replicated. Yeah, if only there were another trial that showed a positive cognitive effect of bioavailable curcumin... https://www.ncbi.nlm.nih.gov/pubmed/25277322 https://www.ncbi.nlm.nih.gov/pubmed/25277322 I mean, the title of this post is by far the most wrong thing about this article, as... 1. it's not turmeric, it's curcumin, which is about 2% of turmeric by mass (so, you'd need to eat ~4.5g turmeric daily to match study doses). 2. curcumin doesn't even have any reasonable bioavilability by itself, so they're using a bioavailable analogue.

andrewjl 9y ago
What prevents researchers from running studies like these on the order of thousands of individuals, as opposed to 40 or 100?
- shanedrgn 9y ago
  Funding
- sov 9y ago
  A lot of things that basically boil down to budget, optimization and space available. If you have a drug that you believe you can show a strong effect with the effect group at ~15 people, maybe you run the trial with 40 (15 e.g., 15 control, plus wiggle room for drop-outs over the 18-month course and extra certainty). Check the last few paragraphs--"The authors thank Shayna Greenberg, Dev Darshan Khalsa, and Anya Rosensteel for help in recruitment, data management, and study coordination; John Williams, University of California, Los Angeles, Nuclear Medicine Clinic, for performing the PET scans; and Vladimir Kepe, Cleveland Clinic, for developing the FDDNP-PET analysis procedures for proteinopathies in patients with neurodegenerative diseases." Add all that to the effort involved in the actual chemical extraction of the bioavailable curcumin analogue and it just doesn't make sense to add any more people than you need.

piracy1 9y ago

It's possible, but I think (not that I'm an expert) that this is not the first piece of evidence to suggest this. A year or two was recently speaking with a researcher who studies Alzheimer's and he had told me about how in cultures where more curry and turmeric was consumed there was significantly less Alzherimer's. I'll try to find a source for this and update my comment.

lstyls 9y ago
Not only is that anecdotal, but even if it's true it is correlative in the weakest sense. There are undoubtedly countless other features for which "cultures" (how does one define this concretely btw) with low Alzheimer's rates also are outliers. Here is a great demonstration of how correlation does not imply causation: http://www.tylervigen.com/spurious-correlations http://www.tylervigen.com/spurious-correlations.
- piracy1 9y ago
  Sure. But I'm saying that bit of information would become interesting given the original post which supports a similar idea. I'm not making an assertion of fact simply bringing to mind another data point. Also here. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2781139/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2781139/ It would seem that this is a rather well documented and often researched thing. Not just a one-off p-hack. I believe the research I was thinking of was specifically this part of the overview above.: Various studies and research[9,10] results indicate a lower incidence and prevalence of AD in India. The prevalence of AD among adults aged 70-79 years in India is 4.4 times less than that of adults aged 70-79 years in the United States.[9] Researchers investigated the association between the curry consumption and cognitive level in 1010 Asians between 60 and 93 years of age. The study found that those who occasionally ate curry (less than once a month) and often (more than once a month) performed better on a standard test (MMSE) of cognitive function than those who ate curry never or rarely.[10] Sources: 9. Pandav R, Belle SH, DeKosky ST. Apolipoprotein E polymorphism and Alzheimer's disease: The Indo-US cross-national dementia study. Arch Neurol. 2000;57:824–30. [PubMed] 10. Ng TP, Chiam PC, Lee T, Chua HC, Lim L, Kua EH. Curry consumption and cognitive function in the elderly. Am J Epidemiol. 2006;164:898–906. [PubMed]

EnderWT 9y ago

Examine.com has a page for curcumin. They cite 5 sources with notable impacts on reducing depression, but they summarize with this: "Curcumin seems to be somewhat more effective than placebo in reducing symptoms of depression. It may take 2-3 months to see any outcomes. Skepticism is warranted though, as the studies comparing curcumin to placebo were not well designed and produced effect sizes not too far apart, even though the differences were statistically significant." https://examine.com/supplements/curcumin/#hem-depression https://examine.com/supplements/curcumin/#hem-depression

roystonvassey 9y ago

This is becoming an increasingly worrying trend - to bunk any analysis with a comment on 'small sample size, biases' that somehow moves to the top of the comment pile. Statistically, there is no such thing as a 'small sample size', in isolation. It is always related to the power and effect that the researchers want to work with. Moreover, even if a study is statistically insignificant, it DOES NOT mean that the opposite conclusion is true. It might often merit some further examination of cultural and historical trends. In this case, for instance, turmeric has been used for medicinal properties in Asian cultures for centuries. Theories of evolution suggest that this wouldn't be the case (and it would've died a natural death), if there wasn't some merit.

carlmr 9y ago
>In this case, for instance, turmeric has been used for medicinal properties in Asian cultures for centuries. Theories of evolution suggest that this wouldn't be the case (and it would've died a natural death), if there wasn't some merit. Lots of things that are completely useless have been used for thousands of years for their medicinal purposes. The evolution argument only holds if there were some negative effects here. Anything that doesn't do harm to the body can easily integrate into the huge body of knowledge called superstition. Eating it is tasty and doesn't hurt, there's nothing more needed to explain that Asian cultures have it as a medicinal herb. If it for example was good for you but caused somewhat unpleasant side-effects, your evolutionary argument might have more merit. Anything that works as well as homeopathy (it doesn't), but doesn't harm either (like homeopathy), won't be sorted out.
afromatic 9y ago
Moreover, even if a property is evolutionarily insignificant, it DOES NOT mean that it will be selected against.
Retric 9y ago
Statistics makes basic assumptions that are false. For example, most things don't fit a bell curve. These assumptions are more relevant in small sample sizes. Making them inherently unreliable. What's missing is not a p-value, but an estimate for the uncertainty around that p-value.

pytonslange 9y ago

It was registered as a clinical trial (https://clinicaltrials.gov/ct2/show/NCT01383161?term=NCT01383161&rank=1 https://clinicaltrials.gov/ct2/show/NCT01383161?term=NCT0138...). If you are concerned about p-hacking and replicability, what should worry you is that the outcome measures posted during registration are rather vague, i.e. ".. show less evidence of cognitive decline (as measured with neuropsychological assessments)" At baseline, they performed an extensive neuropsychologist test battery, including a bunch of subtests: -Trail Making Test A -WAIS-III Digit Symbol Substitution -WAIS-III Block Design Test -Rey-Osterrieth Complex Figure Test (copy) -Trail Making Test B -Stroop Interference -F.A.S. -Buschke-Fuld Selective Reminding Test [SRT] -Wechsler Memory Scale-3rd Edition [WMS-III] -Verbal Paired Associations I -Benton Visual Retention Test -Buschke-Fuld SRT -Rey-Osterrieth Complex Figure Test [recall] -WMS-III Verbal Paired Associations II -Boston Naming Test -Animal Naming Test The outcome measures they report are: -TMT-A -SRT -BVMT-R I would be _very_ surprised if those were the only tests they performed at followup after a 18 month (expensive) clinical trial. If these measures had been specified as the only outcome measures of interest, noone would question the results. But, when it was not registered, and when no other measures are reported, not even in the supplementary, the reader is left to wonder why that is.

closed 9y ago
That's a good point. After having seen much of the brain training literature (especially its early studies), this wouldn't surprise me at all.
hanklazard 9y ago
Excellent point. Outcome measures should have been more clearly defined in pre-registration. As it was done, p-hacking and selective reporting were still possible.

dumbneurologist 9y ago

agree that replication is important and that it's too early to believe fully, but when a study size is small and you still find a difference, then it implies the effect size is large. If you meet statistical significance, then diminishing n means increasing clinical significance.

klmr 9y ago
> but when a study size is small and you still find a difference, then it implies the effect size is large That’s only true if it holds across multiple studies. Otherwise it might simply be random variability. By your argument, a study with n=4 that showed an effect would imply a huge effect size. No. It implies that the measured signal is highly variable, and that you got lucky on 1:3 odds. Check out regression towards the mean [1]. (Note that if you’ve got a good underlying model of the variability, n=4 can work. In fact, it’s routinely done in specific applications, such as transcriptomics. But even there it’s far from ideal, and it’s supported by a stringent error model.) [1] https://en.wikipedia.org/wiki/Regression_toward_the_mean https://en.wikipedia.org/wiki/Regression_toward_the_mean
pbhjpbhj 9y ago
I flipped 5 coins, 4 came up heads, so heads must be >>80% likelihood??

dumbneurologist 9y ago

Except that it was preregistered, as described in the article. https://clinicaltrials.gov/ct2/show/NCT01383161?cond=NCT01383161&rank=1 https://clinicaltrials.gov/ct2/show/NCT01383161?cond=NCT0138... It's great to be skeptical, but it's really lame to be skeptical and lazy. It's a terrible combination that causes patients not to trust their doctors.