4 ms·

The salmon thing was a statistical problem, not a technical one. fMRI data generate an incredible number of data points: imagine a movie, but in three dimensio

The salmon thing was a statistical problem, not a technical one. fMRI data generate an incredible number of data points: imagine a movie, but in three dimensions, so you get a sequence of x/y/z-volumes. A typical scan has ~128 to 256 voxels in each spatial dimension, for ~1 million voxels per volume. This means that if your analysis contains voxel-by-voxel tests, you're going to be running a huge number of them. Even if each test has a fairly low false-positive rate (say 0.1%), there are still a huge number of tests, and thus, a huge number of false positives. There are principled ways of correcting for this; there are also hacky "folk methods" like setting a more stringent false positive threshold. The fish poster argues that the latter doesn't work, using a deliberately silly example. http://prefrontal.org/files/posters/Bennett-Salmon-2009.pdf http://prefrontal.org/files/posters/Bennett-Salmon-2009.pdf

Alex3917 10y ago
> The salmon thing was a statistical problem, not a technical one. How is that relevant to the question of whether or not most fMRI research is wrong?
- mattkrause 10y ago
  1. It's trivially checkable for any existing individual paper. You skim the methods or search for "multiple comparisons" or "false discovery rate" or something like that. 2. For papers where this wasn't done, the already-collected data can be reanalyzed. In fact, you can often get correct it without access to the raw data (at least approximately). 3. It means that future papers (where future is somewhere after 2008-9 here) can be done correctly from the get-go; it's not a limitation of the technique or the signal itself.
  - Alex3917 10y ago
    If you believe that the majority of papers in the field are not incorrect, what are the assumptions behind your estimation of the numbers?
- omginternets 10y ago
  The significance of the dead salmon paper is waaaay overblown. That paper was a warning to researchers that family-wise error corrections are, indeed, important. A small (but non-negligible) portion of researchers were not doing these corrections simply because "the results looked nicer". The dead fish guys wanted to demonstrate the extent to which this could produce false positives. The (rightly) make no comment on the soundness of fMRI as a tool for cognitive science.

Alex3917 10y ago

> The salmon thing was a statistical problem, not a technical one. How is that relevant to the question of whether or not most fMRI research is wrong?

mattkrause 10y ago
1. It's trivially checkable for any existing individual paper. You skim the methods or search for "multiple comparisons" or "false discovery rate" or something like that. 2. For papers where this wasn't done, the already-collected data can be reanalyzed. In fact, you can often get correct it without access to the raw data (at least approximately). 3. It means that future papers (where future is somewhere after 2008-9 here) can be done correctly from the get-go; it's not a limitation of the technique or the signal itself.
- Alex3917 10y ago
  If you believe that the majority of papers in the field are not incorrect, what are the assumptions behind your estimation of the numbers?
omginternets 10y ago
The significance of the dead salmon paper is waaaay overblown. That paper was a warning to researchers that family-wise error corrections are, indeed, important. A small (but non-negligible) portion of researchers were not doing these corrections simply because "the results looked nicer". The dead fish guys wanted to demonstrate the extent to which this could produce false positives. The (rightly) make no comment on the soundness of fMRI as a tool for cognitive science.