4 ms·

Cognitive training: A field in search of a phenomenon

62 points by beefman 4y ago

a9h74j 4y ago
> As is clear from the empirical evidence reviewed in the previous sections, the likelihood that cognitive training provides broad cognitive and academic benefits is very low indeed; therefore, resources should be devoted to other scientific questions—it is not rational to invest considerable sums of money on a scientific question that has been essentially answered by the negative. In a recent article, Green et al. (2019) took the exact opposite of this decision—they strongly recommended that funding agencies should increase funding for cognitive training. This obviously calls for comments. I wonder how the lack of direct results relate to cognitive rehabilitation for persons with some medical etc issue. Could one value be to make it very evident to a person that they do have a deficit and need to work on compensation strategies? So measured by overall well-being cognitive rehabilitation could produce positive measurable outcomes?
- colechristensen 4y ago
  There's a problem I have with this kind of science. It boils down to "We tried to achieve X by doing Y and failed, therefore X isn't possible". Assuming Y is the correct way to do X and then giving up. Instead it should almost be something like a competition. "We tried to achieve X by doing Y_{1..n} and these n had statistically significant effects of these sizes" When you have a lot of people trying to accomplish something, some of them will end up being successful if the thing is possible, but there is an art to finding that optimum and published science really often seems to fall short.
  - thrown_22 4y ago
    We also have Q P S R T U V and W that need funding and aren't getting it because X is getting all of it. If X is too hard try something else that's easier.
  - scott_s 4y ago
    I think you are mischaracterizing this meta-analysis. First, a meta-analysis is never about just trying one thing; that's what a single study is for. A meta-analysis looks at many studies, over a long period of time and tries to find consistent patterns that emerge from the whole. That's how we try to systematically figure out if results are by chance, or if there's a real effect. The pattern these authors are pulling from all of these studies is there is no real effect. When you find no real effect over many studies for a long time, you inevitably have to confront the question: is what we're trying to do possible?
    - mlyle 4y ago
      > When you find no real effect over many studies for a long time, you inevitably have to confront the question: is what we're trying to do possible? Devil's advocate: The same would apply to say, Alzheimer's treatments, where we've had no real effect over many studies for a long time. Should we give up?
      - yababa_y 4y ago
        Lack of progress is a good reason to backtrack up the search tree. Which is exactly what we are seeing with alzheimer’s research. Amyloid hypothesis isn’t working, so backtrack. It’s a logical error to extend that to “give up”
        mlyle 4y ago
        The logical extension of this would be that perhaps it's time to not spend as much effort on some approaches at cognitive training, but not to give up on possibility of the entire idea.
    - colechristensen 4y ago
      >When you find no real effect over many studies for a long time, you inevitably have to confront the question: is what we're trying to do possible? This is what I'm criticizing. The methodology for searching for real effects is broken.
    - 4y ago
paulpauper 4y ago
[flagged]
- keepquestioning 4y ago
light_hue_1 4y ago
Yes. Everyone adjacent to this field has known for a long time that it's just a bunch of charlatans and liars looking to prey on older people and people with disabilities. There are so many startups (like say https://www.neurotrackerx.com/ https://www.neurotrackerx.com/) devoted to taking people's money based on nonsense "science". Where even if the science was true, the effect sizes are so small as to be meaningless.
- torginus 4y ago
  I just wonder how do you get so many prestigious universities, companies, medical institution etc. to endorse your product if it basically doesn't work?
  - mjburgess 4y ago
    There's no real downside. They're part of the scam. I think this largely applies to AI (et al.) too. The universities are the enables of a system of "commercialisable pseudoscience", and are in bed with all those companies profiting from it
    - torginus 4y ago
      Does that mean that I can go to UC Berkeley (it's on the website) with a sufficiently big bag of bills, and they'll endorse my miracle ointment that supposedly cures cancer?
      - gwd 4y ago
        Well, any individual person can conclusively tell if they died of cancer. It's not so easy to conclusively for an individual to tell if their brain is getting generally "fitter", or if they're only getting better at the specific "exercises" they're doing. IOW, if you want to show up with a big bag of bills and expect support, you need something much less falsifiable.
  - pmontra 4y ago
    If there are customers, they sell the product. Most of the stuff we buy everyday isn't as good as in the ads or it does us any good at all (maybe it damages us.)
hackerlight 4y ago
> We demonstrate that this optimism is due to the field neglecting the results of meta-analyses and largely ignoring the statistical explanation that apparent effects are due to a combination of sampling errors and other artifacts. Oh where have I heard this before? Medicine, psychology, economics, and every other high-level empirical discipline that's trying to tease out small effects which ends up chasing its tail as statistical noise and publication bias trick the community into thinking something is going on, which then wastes decades of research effort and public money. Systemic reform into the culture of how empirical social sciences is carried out is really needed.
- tuf7fijrvtc 4y ago
  One of those things is not three of those things. Medicine and Economics I'd argue have far stronger track records than Psychology which bends theory enough to qualify for the Olympics at even the vaguest sign of political pressure. Yielding over and over to LGBT activism killed most of Psychology's credibility.
Madmallard 4y ago
Do you know what actually provides broad cognitive and academic benefits that is of no surprise to the average person? Eating healthy, being social, and aerobic exercise. Wow! So groundbreaking.
- she11c0de 4y ago
  Agreed. I'd add sleep to the list.
hmahncke 4y ago
>> Examples include the ACTIVE trial, commercial brain-training games (e.g., Neuroracer, Lumosity, and BrainHQ), and multidomain training programs (Binder et al., 2016; Buitenweg et al., 2017; Duyck & Op de Beeck, 2019). To date, none of these regimens have shown compelling evidence, or any evidence at all, of training-induced far transfer to either cognitive tests or real-life skills >> ACTIVE showed that cognitive training slowed decline in instrumental activities of daily living [1], and that adaptive computerized speed training in particular reduced at-fault car crashes [2], reduced depressive symptoms [3], and most importantly reduced the incidence of dementia [4]. The NIH is spending tens of millions of dollars on follow-up trials to extend the results. To dismiss ACTIVE in the brief paragraph is...startling. >> We demonstrate that this optimism is due to the field neglecting the results of meta-analyses >> A strong statement from a paper that doesn't seem to cover multiple positive meta-analyses of cognitive training [5, for example]. In my view, if you read a lot of papers in this field (and I do), the pattern is that negative articles generally focus on working memory training and effects on IQ or "generalized cognitive ability" (whatever that is); and positive articles generally focus on neurocognitive measures and real-world functional measures. One reasonable interpretation [and there are many!] is that programs focused on using working memory techniques to improve IQ are not generally effective, and programs using speed/attention training to improve specific aspects of cognitive and real-world performance are effective. Meanwhile, out in the clinical world, cognitive training is now recommended by clinical guidelines from the American Academy of Neurology and the World Health Organization, and offered as a benefit by a dozen Medicare Advantage plans around the country. Disclaimer: I work at BrainHQ, and have published in this field. Further disclaimer, a HN comment isn't an academic article. [1] https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4055506/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4055506/ [2] https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3057872/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3057872/ [3] https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2657170/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2657170/ [4] https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5700828/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5700828/ [5] https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7050567/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7050567/
- tgv 4y ago
  ACTIVE doesn't have much to show for it, does it? Effect sizes are small, and some are almost absent, while there are differences in other factors between the groups with the same magnitude (drinking, cholesterol, diabetes, subject withdrawal, etc.). The calculation of the effect size is also dubious: it assumes the changes are linear and independent of the starting position, and the baseline is not within the group, although I can't see an easy way to get an objective one either. As to the statistics: it's p-values all over, with all of the associated problems. The p-value for article [4] gets just below the 0.05 mark, but only for speed training. They also do two other comparisons, which are nowhere near significant. That is really suspicious, and there's no correction for multiple comparisons. All in all, this short inspection doesn't convince me the OP article has it wrong.
drdrek 4y ago
Its not a case of the scientific community being unaware of this, even the Wikipedia page of "Cognitive training" has the paragraph in the summary: "Scientific investigation into the effectiveness of brain training activities have concluded that they have no impact on intelligence or everyday cognitive ability, and that most programs had no peer reviewed published evidence of their efficacy." Its just a bunch of people that profit from it and no amount of negative research papers will dissuade them as long as there is money to be made.

62 points by beefman 4y ago

> As is clear from the empirical evidence reviewed in the previous sections, the likelihood that cognitive training provides broad cognitive and academic benefits is very low indeed; therefore, resources should be devoted to other scientific questions—it is not rational to invest considerable sums of money on a scientific question that has been essentially answered by the negative. In a recent article, Green et al. (2019) took the exact opposite of this decision—they strongly recommended that funding agencies should increase funding for cognitive training. This obviously calls for comments. I wonder how the lack of direct results relate to cognitive rehabilitation for persons with some medical etc issue. Could one value be to make it very evident to a person that they do have a deficit and need to work on compensation strategies? So measured by overall well-being cognitive rehabilitation could produce positive measurable outcomes?

colechristensen 4y ago
There's a problem I have with this kind of science. It boils down to "We tried to achieve X by doing Y and failed, therefore X isn't possible". Assuming Y is the correct way to do X and then giving up. Instead it should almost be something like a competition. "We tried to achieve X by doing Y_{1..n} and these n had statistically significant effects of these sizes" When you have a lot of people trying to accomplish something, some of them will end up being successful if the thing is possible, but there is an art to finding that optimum and published science really often seems to fall short.
- thrown_22 4y ago
  We also have Q P S R T U V and W that need funding and aren't getting it because X is getting all of it. If X is too hard try something else that's easier.
- scott_s 4y ago
  I think you are mischaracterizing this meta-analysis. First, a meta-analysis is never about just trying one thing; that's what a single study is for. A meta-analysis looks at many studies, over a long period of time and tries to find consistent patterns that emerge from the whole. That's how we try to systematically figure out if results are by chance, or if there's a real effect. The pattern these authors are pulling from all of these studies is there is no real effect. When you find no real effect over many studies for a long time, you inevitably have to confront the question: is what we're trying to do possible?
  - mlyle 4y ago
    > When you find no real effect over many studies for a long time, you inevitably have to confront the question: is what we're trying to do possible? Devil's advocate: The same would apply to say, Alzheimer's treatments, where we've had no real effect over many studies for a long time. Should we give up?
    - yababa_y 4y ago
      Lack of progress is a good reason to backtrack up the search tree. Which is exactly what we are seeing with alzheimer’s research. Amyloid hypothesis isn’t working, so backtrack. It’s a logical error to extend that to “give up”
      - mlyle 4y ago
        The logical extension of this would be that perhaps it's time to not spend as much effort on some approaches at cognitive training, but not to give up on possibility of the entire idea.
  - colechristensen 4y ago
    >When you find no real effect over many studies for a long time, you inevitably have to confront the question: is what we're trying to do possible? This is what I'm criticizing. The methodology for searching for real effects is broken.
  - 4y ago

paulpauper 4y ago

[flagged]

keepquestioning 4y ago

light_hue_1 4y ago

Yes. Everyone adjacent to this field has known for a long time that it's just a bunch of charlatans and liars looking to prey on older people and people with disabilities. There are so many startups (like say https://www.neurotrackerx.com/ https://www.neurotrackerx.com/) devoted to taking people's money based on nonsense "science". Where even if the science was true, the effect sizes are so small as to be meaningless.

torginus 4y ago
I just wonder how do you get so many prestigious universities, companies, medical institution etc. to endorse your product if it basically doesn't work?
- mjburgess 4y ago
  There's no real downside. They're part of the scam. I think this largely applies to AI (et al.) too. The universities are the enables of a system of "commercialisable pseudoscience", and are in bed with all those companies profiting from it
  - torginus 4y ago
    Does that mean that I can go to UC Berkeley (it's on the website) with a sufficiently big bag of bills, and they'll endorse my miracle ointment that supposedly cures cancer?
    - gwd 4y ago
      Well, any individual person can conclusively tell if they died of cancer. It's not so easy to conclusively for an individual to tell if their brain is getting generally "fitter", or if they're only getting better at the specific "exercises" they're doing. IOW, if you want to show up with a big bag of bills and expect support, you need something much less falsifiable.
- pmontra 4y ago
  If there are customers, they sell the product. Most of the stuff we buy everyday isn't as good as in the ads or it does us any good at all (maybe it damages us.)

hackerlight 4y ago

> We demonstrate that this optimism is due to the field neglecting the results of meta-analyses and largely ignoring the statistical explanation that apparent effects are due to a combination of sampling errors and other artifacts. Oh where have I heard this before? Medicine, psychology, economics, and every other high-level empirical discipline that's trying to tease out small effects which ends up chasing its tail as statistical noise and publication bias trick the community into thinking something is going on, which then wastes decades of research effort and public money. Systemic reform into the culture of how empirical social sciences is carried out is really needed.

tuf7fijrvtc 4y ago
One of those things is not three of those things. Medicine and Economics I'd argue have far stronger track records than Psychology which bends theory enough to qualify for the Olympics at even the vaguest sign of political pressure. Yielding over and over to LGBT activism killed most of Psychology's credibility.

Madmallard 4y ago

Do you know what actually provides broad cognitive and academic benefits that is of no surprise to the average person? Eating healthy, being social, and aerobic exercise. Wow! So groundbreaking.

she11c0de 4y ago
Agreed. I'd add sleep to the list.

hmahncke 4y ago

>> Examples include the ACTIVE trial, commercial brain-training games (e.g., Neuroracer, Lumosity, and BrainHQ), and multidomain training programs (Binder et al., 2016; Buitenweg et al., 2017; Duyck & Op de Beeck, 2019). To date, none of these regimens have shown compelling evidence, or any evidence at all, of training-induced far transfer to either cognitive tests or real-life skills >> ACTIVE showed that cognitive training slowed decline in instrumental activities of daily living [1], and that adaptive computerized speed training in particular reduced at-fault car crashes [2], reduced depressive symptoms [3], and most importantly reduced the incidence of dementia [4]. The NIH is spending tens of millions of dollars on follow-up trials to extend the results. To dismiss ACTIVE in the brief paragraph is...startling. >> We demonstrate that this optimism is due to the field neglecting the results of meta-analyses >> A strong statement from a paper that doesn't seem to cover multiple positive meta-analyses of cognitive training [5, for example]. In my view, if you read a lot of papers in this field (and I do), the pattern is that negative articles generally focus on working memory training and effects on IQ or "generalized cognitive ability" (whatever that is); and positive articles generally focus on neurocognitive measures and real-world functional measures. One reasonable interpretation [and there are many!] is that programs focused on using working memory techniques to improve IQ are not generally effective, and programs using speed/attention training to improve specific aspects of cognitive and real-world performance are effective. Meanwhile, out in the clinical world, cognitive training is now recommended by clinical guidelines from the American Academy of Neurology and the World Health Organization, and offered as a benefit by a dozen Medicare Advantage plans around the country. Disclaimer: I work at BrainHQ, and have published in this field. Further disclaimer, a HN comment isn't an academic article. [1] https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4055506/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4055506/ [2] https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3057872/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3057872/ [3] https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2657170/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2657170/ [4] https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5700828/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5700828/ [5] https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7050567/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7050567/

tgv 4y ago
ACTIVE doesn't have much to show for it, does it? Effect sizes are small, and some are almost absent, while there are differences in other factors between the groups with the same magnitude (drinking, cholesterol, diabetes, subject withdrawal, etc.). The calculation of the effect size is also dubious: it assumes the changes are linear and independent of the starting position, and the baseline is not within the group, although I can't see an easy way to get an objective one either. As to the statistics: it's p-values all over, with all of the associated problems. The p-value for article [4] gets just below the 0.05 mark, but only for speed training. They also do two other comparisons, which are nowhere near significant. That is really suspicious, and there's no correction for multiple comparisons. All in all, this short inspection doesn't convince me the OP article has it wrong.

drdrek 4y ago

Its not a case of the scientific community being unaware of this, even the Wikipedia page of "Cognitive training" has the paragraph in the summary: "Scientific investigation into the effectiveness of brain training activities have concluded that they have no impact on intelligence or everyday cognitive ability, and that most programs had no peer reviewed published evidence of their efficacy." Its just a bunch of people that profit from it and no amount of negative research papers will dissuade them as long as there is money to be made.