4 ms·

There have been large problems with fMRI studies for a long time, even leaving aside the potentially sketchy coupling between the BOLD signal and actual neural

There have been large problems with fMRI studies for a long time, even leaving aside the potentially sketchy coupling between the BOLD signal and actual neural activity and the difficulties accounting for movement artifact. This paper published in 2016 suggests that commonly used statistical packages for analysis of fMRI data can have a false discovery rate of up to 70%: https://www.pnas.org/content/113/28/7900 https://www.pnas.org/content/113/28/7900 More fun, this poster presents the results of fMRI in dead salmon given an open ended mentalising task: https://www.psychology.mcmaster.ca/bennett/psy710/readings/BennettDeadSalmon.pdf https://www.psychology.mcmaster.ca/bennett/psy710/readings/B...

jointpdf 6y ago
> Subject: One mature Atlantic Salmon (Salmo salar) participated in the fMRI study. The salmon was approximately 18 inches long, weighed 3.8 lbs, and was not alive at the time of scanning. > Task: The task administered to the salmon involved completing an open-ended mentalizing task. The salmon was shown a series of photographs depicting human individuals in social situations with a specified emotional valence. The salmon was asked to determine what emotion the individual in the photo must have been experiencing. This is just so, so good. Thank you to this salmon for participating in science.
- SubiculumCode 6y ago
  The salmon experiment was ages ago, and just showed that the threshold for statistical significance was too liberal. At that time, when neuroimaging was quite new, it was common practice to just pick a threshold, say .001, and not do corrections for multiple tests. That proved to be too liberal. It wasn't a totally stupid guess though. Much of the brain's BOLD activity is correlated, so Bonferroni correction was stupidly conservative (not independent tests). It was only then that newer techniques (cluster-based correction) were adopted, which explicitly made an assumption about the spatial autocorrelation of the data. The one assumption of this technique was that this autocorrelation was the same throughout the brain, which ended up not being a good assumption, though computationally useful. Newer methods today try to model the autocorrelation in the data for cluster based correction, or have moved to other numerical techniques for correction of multiple comparisons re repeated resampling. So, while the dead salmon experiment was eye opening to some when the neuroimaging field was very new, it is no longer relevant now. Neuroimaging techniques are constantly improving, both from better analytical techniques and knowledge about the factors that can compromise the veracity of analyses, but they, including the present paper, show that neuroimgaging is bunk science or useless, which is most certainly incorrect.
  - jointpdf 6y ago
    Thanks for this, it was interesting and helpful. My comment was mostly appreciating the sardonic and dry humor in the poster (and the concept of the study itself). It was my first time hearing about this, and it struck me as the type of funny anecdote that could be slipped into a talk/lecture (I find it takes quite a bit of finesse to keep non-technical people engaged when talking about statistics and data science, so humor and relatable examples always help). So, it’s good to know the broader context.
9214 6y ago
Salmon story is more than a decade old at this point, and still as relevant as before; became an instant classic in my neuroscience-related MS programme. This Wired article elaborates on the motivation behind the study: https://www.wired.com/2009/09/fmrisalmon/ https://www.wired.com/2009/09/fmrisalmon/
- SubiculumCode 6y ago
  How is it relevant now? Please expand. Sure its relevant in the sense that fmri requires multiple tests, and that has to be accounted for correctly, but we do so now...with greater and greater accuracy as newer and newer techniques take hold, mind you, for a statistical problem that is very difficult to model. The salmon experiment showed that the thresholds scientists used at the BEGINNING of neuroimaging science were insufficient...this is a new field, methods take time to develop.The salmon experiment is not relevant now because we are not making the same mistakes we made then, over a decade ago.
  - 9214 6y ago
    My comment about relevancy was more in line with @jdoliner's in the neighboring thread. I totally agree that tools and practices weren't quite there at the time and that things are different now. The takeaway from Salmon experiment IMO is not specific to fMRI: you cannot trust a single technique or field "golden" standard (whether flawed or not), and should rather pursue accuracy across different pre- post-processing methods. That's not realistically doable single-handily (at least in my experience), but in the current age it should become the norm to rely on publically available pipelines, probably with some sort of configuration files or even GUIs. I personally worked with fMRI only sparingly, so cannot comment on that in the context of the posted article.
phreeza 6y ago
I think Jack Gallant and colleagues get it right by basically applying the method that works well in machine learning. Train your model on one dataset, and then validate on a held out test set. That should get rid of the majority of all statistical artifacts (as long as you do the split right of course).
LargoLasskhyfv 6y ago
I'll just link to a comment I made about 3 months ago, regarding this: https://news.ycombinator.com/item?id=23418356 https://news.ycombinator.com/item?id=23418356

jointpdf 6y ago

> Subject: One mature Atlantic Salmon (Salmo salar) participated in the fMRI study. The salmon was approximately 18 inches long, weighed 3.8 lbs, and was not alive at the time of scanning. > Task: The task administered to the salmon involved completing an open-ended mentalizing task. The salmon was shown a series of photographs depicting human individuals in social situations with a specified emotional valence. The salmon was asked to determine what emotion the individual in the photo must have been experiencing. This is just so, so good. Thank you to this salmon for participating in science.

SubiculumCode 6y ago
The salmon experiment was ages ago, and just showed that the threshold for statistical significance was too liberal. At that time, when neuroimaging was quite new, it was common practice to just pick a threshold, say .001, and not do corrections for multiple tests. That proved to be too liberal. It wasn't a totally stupid guess though. Much of the brain's BOLD activity is correlated, so Bonferroni correction was stupidly conservative (not independent tests). It was only then that newer techniques (cluster-based correction) were adopted, which explicitly made an assumption about the spatial autocorrelation of the data. The one assumption of this technique was that this autocorrelation was the same throughout the brain, which ended up not being a good assumption, though computationally useful. Newer methods today try to model the autocorrelation in the data for cluster based correction, or have moved to other numerical techniques for correction of multiple comparisons re repeated resampling. So, while the dead salmon experiment was eye opening to some when the neuroimaging field was very new, it is no longer relevant now. Neuroimaging techniques are constantly improving, both from better analytical techniques and knowledge about the factors that can compromise the veracity of analyses, but they, including the present paper, show that neuroimgaging is bunk science or useless, which is most certainly incorrect.
- jointpdf 6y ago
  Thanks for this, it was interesting and helpful. My comment was mostly appreciating the sardonic and dry humor in the poster (and the concept of the study itself). It was my first time hearing about this, and it struck me as the type of funny anecdote that could be slipped into a talk/lecture (I find it takes quite a bit of finesse to keep non-technical people engaged when talking about statistics and data science, so humor and relatable examples always help). So, it’s good to know the broader context.

9214 6y ago

Salmon story is more than a decade old at this point, and still as relevant as before; became an instant classic in my neuroscience-related MS programme. This Wired article elaborates on the motivation behind the study: https://www.wired.com/2009/09/fmrisalmon/ https://www.wired.com/2009/09/fmrisalmon/

SubiculumCode 6y ago
How is it relevant now? Please expand. Sure its relevant in the sense that fmri requires multiple tests, and that has to be accounted for correctly, but we do so now...with greater and greater accuracy as newer and newer techniques take hold, mind you, for a statistical problem that is very difficult to model. The salmon experiment showed that the thresholds scientists used at the BEGINNING of neuroimaging science were insufficient...this is a new field, methods take time to develop.The salmon experiment is not relevant now because we are not making the same mistakes we made then, over a decade ago.
- 9214 6y ago
  My comment about relevancy was more in line with @jdoliner's in the neighboring thread. I totally agree that tools and practices weren't quite there at the time and that things are different now. The takeaway from Salmon experiment IMO is not specific to fMRI: you cannot trust a single technique or field "golden" standard (whether flawed or not), and should rather pursue accuracy across different pre- post-processing methods. That's not realistically doable single-handily (at least in my experience), but in the current age it should become the norm to rely on publically available pipelines, probably with some sort of configuration files or even GUIs. I personally worked with fMRI only sparingly, so cannot comment on that in the context of the posted article.

phreeza 6y ago

I think Jack Gallant and colleagues get it right by basically applying the method that works well in machine learning. Train your model on one dataset, and then validate on a held out test set. That should get rid of the majority of all statistical artifacts (as long as you do the split right of course).

LargoLasskhyfv 6y ago

I'll just link to a comment I made about 3 months ago, regarding this: https://news.ycombinator.com/item?id=23418356 https://news.ycombinator.com/item?id=23418356