7 ms·

Risk of false positives in fMRI of post-mortem Atlantic salmon (2010) [pdf]

113 points by zitterbewegung 9y ago

techdragon 9y ago
This is and remains one of my all time favourite scientific research papers. Understandable, obvious, and not just relevant, actually providing a meaningful contribution to their field by virtue of quantifying existing techniques as not sufficient to fully eliminate spurious data from the 'noise floor' inherent in fMRI machine data.
pmoriarty 9y ago
"Task: The task administered to the salmon involved completing an open-ended mentalizing task. The salmon was shown a series of photographs depicting human individuals in social situations with a specified emotional valence. The salmon was asked to determine what emotion the individual in the photo must have been experiencing."
- solomatov 9y ago
  This is absolutely hilarious!
- jxramos 9y ago
  I'm thinking https://en.wikipedia.org/wiki/Weekend_at_Bernie's https://en.wikipedia.org/wiki/Weekend_at_Bernie's Salmon style!
  - aaron695 9y ago
    Slight spoiler, but if you liked this movie watch Derren Brown - Pushed to the Edge. (It's very educational and fun)
    - Luc 9y ago
      'This is a true story' - first scene of Fargo. (it's not a true story)
      - js2 9y ago
        "A body is found in the frozen North Dakota woods. The cops say the dead Japanese woman was looking for the $1m she saw buried in the film Fargo. But the story didn't end there." https://www.theguardian.com/culture/2003/jun/06/artsfeatures1 https://www.theguardian.com/culture/2003/jun/06/artsfeatures...
prefrontal 9y ago
Hey all. I’m the author of the poster and subsequent paper. Happy to answer any questions that you might have. Glad to see this on Hacker News so many years after the poster first went up!
- pmoriarty 9y ago
  Have you written anything else that's even half as funny?
  - prefrontal 9y ago
    I would love to say that I have, but I fear that I may have peaked in 2010.
- prefrontal 9y ago
  PDF of the paper: https://www.dropbox.com/s/o7gb13j26v9f4dp/Bennett-Salmon-2010.pdf?dl=0 https://www.dropbox.com/s/o7gb13j26v9f4dp/Bennett-Salmon-201...
- wolfram74 9y ago
  I presume someone else has already thought to share with you zach weinersmith's interpretation of your results http://www.smbc-comics.com/comic/fmri http://www.smbc-comics.com/comic/fmri
  - prefrontal 9y ago
    Yeah! One of the other authors on the paper sent it my way right after it was published. Freaking amazing - I love it.
- ars 9y ago
  Did it help? Did people understand the message you were sending and take corrective action?
  - prefrontal 9y ago
loeg 9y ago
I assume this was posted in response to the recent fMRI study from the front page: https://news.ycombinator.com/item?id=15597068 https://news.ycombinator.com/item?id=15597068
- zitterbewegung 9y ago
  Yes you are correct :) it reminded me of that paper.
rav 9y ago
The linked Dropbox page appears to be password-protected. Here's the original poster: http://prefrontal.org/files/posters/Bennett-Salmon-2009.pdf http://prefrontal.org/files/posters/Bennett-Salmon-2009.pdf
- prefrontal 9y ago
  I think it was just asking you to register for Dropbox. If you close that popup the PDF is already open behind it. Here's a link from my server that should just work: http://prefrontal.org/files/papers/Bennett-Salmon-2010.pdf http://prefrontal.org/files/papers/Bennett-Salmon-2010.pdf
  - rav 9y ago
    Huh! When I click "Cancel", the PDF disappears and then there's just a centered box saying "This file is password-protected. Bennett-Salmon-2010.pdf · 0.94 MB". However, the Download link works just fine and lets me download and read the PDF. Dropbox is weird.
omginternets 9y ago
Ahh such a classic :) tl;dr: correcting for multiple comparisons is important.
_qbxp 9y ago
I was doing fMRI work around the time this paper was published. It astonished me that people would simply set an uncorrected voxel-level threshold and call it a day. No FWE-correction, no cluster-threshold - just an 0.001 uncorrected threshold. It was sad that this paper needed to be published to get researchers to start paying attention to that. I'll be honest - when the paper was published I was thinking "no shit - why do we need a paper to tell us what we all learned in stats 101 about multiple comparisons??" And then realized the quantity of fMRI papers that used uncorrected thresholds. Very similar feeling when the "Voodoo Correlations" paper came out. Except I was admittedly guilty of having presented correlation coefficients from clusters that had already been identified using thresholding. So that paper really did make me take a closer look at some of my figures/conclusions.
- pygy_ 9y ago
  I was unaware of the salmon paper, but I remember a bit later being very puzzled by a study about comatose/vegetative patients that included brain dead subjects as controls (and off course healthy subjects as well). I suppose it was meant to convince statistically naive readers that the dead salmon thing didn't apply to their methodology.
- SubiculumCode 9y ago
  Let us be a little bit fair to the researchers who adopted the p=.001 or p=.0001 uncorrected approaches. Their approach wasn't completely unreasoned, and was even justifiable at one time given available methods. There were mainly two approaches to multiple comparison corrections: Bonferroni and setting an uncorrected threshold. People here might say, well yeah, use Bonferroni. However, Bonferroni is really only appropriate when comparisons are independent. Voxels (3D pixels) which are adjacent are highly dependent, and indeed the brain is generally correlated. This dependency makes Bonferrnoi correction (very) inappropriately conservative. Given the average dependence of voxels, some researchers estimated that the average number of true comparisons might be on the order of hundreds to a few thousand. In practice, researchers corrected with Bonferroni, either found a really strong effect, or reset using uncorrected threshold. Some reported results using both. People who read the results interpreted results that way too. Bonferroni = reliable, uncorrected = provisional The contribution of the salmon study and other research papers is that they truly demonstrated that the typical uncorrected thresholds in use were insufficient to control false positives.

This is and remains one of my all time favourite scientific research papers. Understandable, obvious, and not just relevant, actually providing a meaningful contribution to their field by virtue of quantifying existing techniques as not sufficient to fully eliminate spurious data from the 'noise floor' inherent in fMRI machine data.

pmoriarty 9y ago

"Task: The task administered to the salmon involved completing an open-ended mentalizing task. The salmon was shown a series of photographs depicting human individuals in social situations with a specified emotional valence. The salmon was asked to determine what emotion the individual in the photo must have been experiencing."

solomatov 9y ago
This is absolutely hilarious!
jxramos 9y ago
I'm thinking https://en.wikipedia.org/wiki/Weekend_at_Bernie's https://en.wikipedia.org/wiki/Weekend_at_Bernie's Salmon style!
- aaron695 9y ago
  Slight spoiler, but if you liked this movie watch Derren Brown - Pushed to the Edge. (It's very educational and fun)
  - Luc 9y ago
    'This is a true story' - first scene of Fargo. (it's not a true story)
    - js2 9y ago
      "A body is found in the frozen North Dakota woods. The cops say the dead Japanese woman was looking for the $1m she saw buried in the film Fargo. But the story didn't end there." https://www.theguardian.com/culture/2003/jun/06/artsfeatures1 https://www.theguardian.com/culture/2003/jun/06/artsfeatures...

prefrontal 9y ago

Hey all. I’m the author of the poster and subsequent paper. Happy to answer any questions that you might have. Glad to see this on Hacker News so many years after the poster first went up!

pmoriarty 9y ago
Have you written anything else that's even half as funny?
- prefrontal 9y ago
  I would love to say that I have, but I fear that I may have peaked in 2010.
prefrontal 9y ago
PDF of the paper: https://www.dropbox.com/s/o7gb13j26v9f4dp/Bennett-Salmon-2010.pdf?dl=0 https://www.dropbox.com/s/o7gb13j26v9f4dp/Bennett-Salmon-201...
wolfram74 9y ago
I presume someone else has already thought to share with you zach weinersmith's interpretation of your results http://www.smbc-comics.com/comic/fmri http://www.smbc-comics.com/comic/fmri
- prefrontal 9y ago
  Yeah! One of the other authors on the paper sent it my way right after it was published. Freaking amazing - I love it.
ars 9y ago
Did it help? Did people understand the message you were sending and take corrective action?
- prefrontal 9y ago

loeg 9y ago

I assume this was posted in response to the recent fMRI study from the front page: https://news.ycombinator.com/item?id=15597068 https://news.ycombinator.com/item?id=15597068

zitterbewegung 9y ago
Yes you are correct :) it reminded me of that paper.

rav 9y ago

The linked Dropbox page appears to be password-protected. Here's the original poster: http://prefrontal.org/files/posters/Bennett-Salmon-2009.pdf http://prefrontal.org/files/posters/Bennett-Salmon-2009.pdf

prefrontal 9y ago
I think it was just asking you to register for Dropbox. If you close that popup the PDF is already open behind it. Here's a link from my server that should just work: http://prefrontal.org/files/papers/Bennett-Salmon-2010.pdf http://prefrontal.org/files/papers/Bennett-Salmon-2010.pdf
- rav 9y ago
  Huh! When I click "Cancel", the PDF disappears and then there's just a centered box saying "This file is password-protected. Bennett-Salmon-2010.pdf · 0.94 MB". However, the Download link works just fine and lets me download and read the PDF. Dropbox is weird.

omginternets 9y ago

Ahh such a classic :) tl;dr: correcting for multiple comparisons is important.

_qbxp 9y ago

I was doing fMRI work around the time this paper was published. It astonished me that people would simply set an uncorrected voxel-level threshold and call it a day. No FWE-correction, no cluster-threshold - just an 0.001 uncorrected threshold. It was sad that this paper needed to be published to get researchers to start paying attention to that. I'll be honest - when the paper was published I was thinking "no shit - why do we need a paper to tell us what we all learned in stats 101 about multiple comparisons??" And then realized the quantity of fMRI papers that used uncorrected thresholds. Very similar feeling when the "Voodoo Correlations" paper came out. Except I was admittedly guilty of having presented correlation coefficients from clusters that had already been identified using thresholding. So that paper really did make me take a closer look at some of my figures/conclusions.

pygy_ 9y ago
I was unaware of the salmon paper, but I remember a bit later being very puzzled by a study about comatose/vegetative patients that included brain dead subjects as controls (and off course healthy subjects as well). I suppose it was meant to convince statistically naive readers that the dead salmon thing didn't apply to their methodology.
SubiculumCode 9y ago
Let us be a little bit fair to the researchers who adopted the p=.001 or p=.0001 uncorrected approaches. Their approach wasn't completely unreasoned, and was even justifiable at one time given available methods. There were mainly two approaches to multiple comparison corrections: Bonferroni and setting an uncorrected threshold. People here might say, well yeah, use Bonferroni. However, Bonferroni is really only appropriate when comparisons are independent. Voxels (3D pixels) which are adjacent are highly dependent, and indeed the brain is generally correlated. This dependency makes Bonferrnoi correction (very) inappropriately conservative. Given the average dependence of voxels, some researchers estimated that the average number of true comparisons might be on the order of hundreds to a few thousand. In practice, researchers corrected with Bonferroni, either found a really strong effect, or reset using uncorrected threshold. Some reported results using both. People who read the results interpreted results that way too. Bonferroni = reliable, uncorrected = provisional The contribution of the salmon study and other research papers is that they truly demonstrated that the typical uncorrected thresholds in use were insufficient to control false positives.