5 ms·

It's already been known for several years that almost all MRI brain scan research is wrong, what exactly is new here?

by Alex3917 10y ago

JumpCrisscross 10y ago
> almost all MRI brain scan research is wrong Source for a layman?
- goalieca 10y ago
  I wasn't in fmri research but some fellow students in my lab were on it. I do know there was a study with dead salmon which were showing up as having "active" brain regions. I have to find it but am on my phone. It's easy to imagine the main difficulty with this is mapping signal to actual thoughts and regions. It's a really complex biology and physics to reduce.
  - leemailll 10y ago
    what is your field? Neuroscience?
    - goalieca 10y ago
      Electrical engineering (medical imaging - medical imaging technologies) This is the best I could find for the paper. not what I originally meant to cite. http://onlinelibrary.wiley.com/doi/10.1002/scin.5591761320/abstract http://onlinelibrary.wiley.com/doi/10.1002/scin.5591761320/a...
  - greenyoda 10y ago
    Searching for "dead salmon fMRI" turned up the following: http://www.prefrontal.org/files/posters/Bennett-Salmon-2009.pdf http://www.prefrontal.org/files/posters/Bennett-Salmon-2009....
    - acveilleux 10y ago
      Back when I was involved in MRI research, multiple comparisons correction was the rock that many a paper in the field crashed upon. Neuro psych. suffers from weak stats, which is a far cry from saying either fMRI itself is meaningless or saying MRI as a whole is.
  - deleted 10y ago
    [deleted]
  - mattkrause 10y ago
    The salmon thing was a statistical problem, not a technical one. fMRI data generate an incredible number of data points: imagine a movie, but in three dimensions, so you get a sequence of x/y/z-volumes. A typical scan has ~128 to 256 voxels in each spatial dimension, for ~1 million voxels per volume. This means that if your analysis contains voxel-by-voxel tests, you're going to be running a huge number of them. Even if each test has a fairly low false-positive rate (say 0.1%), there are still a huge number of tests, and thus, a huge number of false positives. There are principled ways of correcting for this; there are also hacky "folk methods" like setting a more stringent false positive threshold. The fish poster argues that the latter doesn't work, using a deliberately silly example. http://prefrontal.org/files/posters/Bennett-Salmon-2009.pdf http://prefrontal.org/files/posters/Bennett-Salmon-2009.pdf
    - Alex3917 10y ago
      > The salmon thing was a statistical problem, not a technical one. How is that relevant to the question of whether or not most fMRI research is wrong?
      - mattkrause 10y ago
        1. It's trivially checkable for any existing individual paper. You skim the methods or search for "multiple comparisons" or "false discovery rate" or something like that. 2. For papers where this wasn't done, the already-collected data can be reanalyzed. In fact, you can often get correct it without access to the raw data (at least approximately). 3. It means that future papers (where future is somewhere after 2008-9 here) can be done correctly from the get-go; it's not a limitation of the technique or the signal itself.
        Alex3917 10y ago
        If you believe that the majority of papers in the field are not incorrect, what are the assumptions behind your estimation of the numbers?
      - omginternets 10y ago
- Alex3917 10y ago
  Here are a couple Ben Goldacre articles on neuroscience in general: https://www.theguardian.com/commentisfree/2011/sep/09/bad-science-research-error https://www.theguardian.com/commentisfree/2011/sep/09/bad-sc... For fMRI research specifically, I seem to remember that almost all of the analysis software is closed source, and that comparing the readings between software versions isn't actually meaningful even though a lot of papers do so.
  - acveilleux 10y ago
    That's actually the same article on two sites. Itself just a layman's piece on an actual article in Nature Neuroscience.
    - Alex3917 10y ago
      Oops, deleted the second link. The reason I didn't link to the original journal article is that JumpCrisscross specifically asked for a source for a layman.
  - leemailll 10y ago
    Actually for fMRI studies most people use matlab and share it. It is not really closed-sourced. But I guess many guys in the field won't bother too much to read the codes from a widely used and well-cited matlab script
    - deleted 10y ago
      [deleted]
  - mattkrause 10y ago
    > almost all of the analysis software is closed source This is almost certainly not true. BrainSurfer is closed-source, but almost everything else I can think of is open-source. * SPM is available under the GPL: http://www.fil.ion.ucl.ac.uk/spm/software/ http://www.fil.ion.ucl.ac.uk/spm/software/ * FSL (including the source) is available under their own noncommercial "As-Is" license. FSLView seems to have its own separate GPL license: http://fsl.fmrib.ox.ac.uk/fsldownloads/ http://fsl.fmrib.ox.ac.uk/fsldownloads/ * AFNI used to require (free) registration, but is now GPL: https://afni.nimh.nih.gov/afni/about/legal https://afni.nimh.nih.gov/afni/about/legal * The BIC-MNI tools + source appear to be freely available for free under an "as-is" + attribution license. I only checked a few; please let me know if any are not and I will nag people accordingly: https://github.com/BIC-MNI https://github.com/BIC-MNI * Caret is GPL: http://brainvis.wustl.edu/wiki/index.php/Caret:About http://brainvis.wustl.edu/wiki/index.php/Caret:About * FreeSurfer has its own license which is very open: http://www.freesurfer.net/fswiki/FreeSurferSoftwareLicense http://www.freesurfer.net/fswiki/FreeSurferSoftwareLicense I suspect you're thinking of this, which reported differences between FreeSurfer versions: http://journals.plos.org/plosone/article/comment?id=info%3Adoi%2F10.1371%2Fannotation%2F2054bfcc-614f-4fe0-b1e4-72b6e89ba8e5 http://journals.plos.org/plosone/article/comment?id=info%3Ad... That turned out to be a combination of 1) algorithmic changes 2) Different random seeds (some of these functions use Monte Carlo-like methods), and 3) 32 vs 64 bit platforms.
verytrivial 10y ago
Citation?
op00to 10y ago
It's not that the research is 100% wrong, but the way the research is reported in the news is generally wrong. No, researchers can't really tell if you're thinking about a sailboat from looking at an fMRI, despite whatever the newspaper says. Source: worked in an fMRI research lab for 4 years.
acveilleux 10y ago
There's a fair bit of junky data and quite a few analysis methods that cannot compare data accurately between magnets, position wrt isocentre, or even processing software revisions. But your claim is a lot bolder than that.
- Alex3917 10y ago
  > There's a fair bit of junky data and quite a few analysis methods that cannot compare data accurately between magnets, position wrt isocentre, or even processing software revisions. But your claim is a lot bolder than that. What is the prevalence of each of those types of errors? Since AFAIK many of those errors occur in 30%+ of papers, unless you're assuming close to 0% independence, my claim doesn't seem especially bold...
  - acveilleux 10y ago
    The methods I am familiar with, mostly dealing with volumetric measurements, registration, and automated segmentation were either robust or had well characterized limitations. I take some offense to blanketing the entire field of MRI based on a couple articles pointing out that some (admittedly a lot) fMRI experiments are statistically unsound.
    - Alex3917 10y ago
      How would you estimate the prevalence of inaccurate research in the field, if not by multiplying each (known) source of error by its estimated prevalence, and also using some estimate of the independence between errors? (And of course including general sources of error that affect all scientific research and aren't specific to any particular field.) And as far as independence goes, I haven't seen any research to suggest these errors have anything less than 100% independence, although if there is any research suggesting that these errors (or any other errors in science research) are correlated then I'd love to see that.
mattkrause 10y ago
There's this weird snobbishness about fMRI: it's uniquely terrible, the people are hacks, etc. It seems particularly common amongst first and second-year grad students who are doing something they think is "harder" science. I hate it. In the right hands, fMRI can be a really powerful technique for probing neural activity in healthy human subjects; in fact, it's one of the only ways to do so that has decent spatial resolution and thus lets you link brain structure and function. It certainly does have problems. There are plenty of ways to subtly mess up the data analysis or over-interpret results. The experimental design is often lacking, etc. However, I think these largely reflect the very low barrier of entry to fMRI research--all you really need is a laptop and somewhere willing to sell you scanner time (almost all major universities or hospitals)--rather than some intrinsic limitation of the field. The good work remains very good.

It's already been known for several years that almost all MRI brain scan research is wrong, what exactly is new here?

JumpCrisscross 10y ago

> almost all MRI brain scan research is wrong Source for a layman?

goalieca 10y ago
I wasn't in fmri research but some fellow students in my lab were on it. I do know there was a study with dead salmon which were showing up as having "active" brain regions. I have to find it but am on my phone. It's easy to imagine the main difficulty with this is mapping signal to actual thoughts and regions. It's a really complex biology and physics to reduce.
- leemailll 10y ago
  what is your field? Neuroscience?
  - goalieca 10y ago
    Electrical engineering (medical imaging - medical imaging technologies) This is the best I could find for the paper. not what I originally meant to cite. http://onlinelibrary.wiley.com/doi/10.1002/scin.5591761320/abstract http://onlinelibrary.wiley.com/doi/10.1002/scin.5591761320/a...
- greenyoda 10y ago
  Searching for "dead salmon fMRI" turned up the following: http://www.prefrontal.org/files/posters/Bennett-Salmon-2009.pdf http://www.prefrontal.org/files/posters/Bennett-Salmon-2009....
  - acveilleux 10y ago
    Back when I was involved in MRI research, multiple comparisons correction was the rock that many a paper in the field crashed upon. Neuro psych. suffers from weak stats, which is a far cry from saying either fMRI itself is meaningless or saying MRI as a whole is.
- deleted 10y ago
  [deleted]
- mattkrause 10y ago
  The salmon thing was a statistical problem, not a technical one. fMRI data generate an incredible number of data points: imagine a movie, but in three dimensions, so you get a sequence of x/y/z-volumes. A typical scan has ~128 to 256 voxels in each spatial dimension, for ~1 million voxels per volume. This means that if your analysis contains voxel-by-voxel tests, you're going to be running a huge number of them. Even if each test has a fairly low false-positive rate (say 0.1%), there are still a huge number of tests, and thus, a huge number of false positives. There are principled ways of correcting for this; there are also hacky "folk methods" like setting a more stringent false positive threshold. The fish poster argues that the latter doesn't work, using a deliberately silly example. http://prefrontal.org/files/posters/Bennett-Salmon-2009.pdf http://prefrontal.org/files/posters/Bennett-Salmon-2009.pdf
  - Alex3917 10y ago
    > The salmon thing was a statistical problem, not a technical one. How is that relevant to the question of whether or not most fMRI research is wrong?
    - mattkrause 10y ago
      1. It's trivially checkable for any existing individual paper. You skim the methods or search for "multiple comparisons" or "false discovery rate" or something like that. 2. For papers where this wasn't done, the already-collected data can be reanalyzed. In fact, you can often get correct it without access to the raw data (at least approximately). 3. It means that future papers (where future is somewhere after 2008-9 here) can be done correctly from the get-go; it's not a limitation of the technique or the signal itself.
      - Alex3917 10y ago
        If you believe that the majority of papers in the field are not incorrect, what are the assumptions behind your estimation of the numbers?
    - omginternets 10y ago
Alex3917 10y ago
Here are a couple Ben Goldacre articles on neuroscience in general: https://www.theguardian.com/commentisfree/2011/sep/09/bad-science-research-error https://www.theguardian.com/commentisfree/2011/sep/09/bad-sc... For fMRI research specifically, I seem to remember that almost all of the analysis software is closed source, and that comparing the readings between software versions isn't actually meaningful even though a lot of papers do so.
- acveilleux 10y ago
  That's actually the same article on two sites. Itself just a layman's piece on an actual article in Nature Neuroscience.
  - Alex3917 10y ago
    Oops, deleted the second link. The reason I didn't link to the original journal article is that JumpCrisscross specifically asked for a source for a layman.
- leemailll 10y ago
  Actually for fMRI studies most people use matlab and share it. It is not really closed-sourced. But I guess many guys in the field won't bother too much to read the codes from a widely used and well-cited matlab script
  - deleted 10y ago
    [deleted]
- mattkrause 10y ago
  > almost all of the analysis software is closed source This is almost certainly not true. BrainSurfer is closed-source, but almost everything else I can think of is open-source. * SPM is available under the GPL: http://www.fil.ion.ucl.ac.uk/spm/software/ http://www.fil.ion.ucl.ac.uk/spm/software/ * FSL (including the source) is available under their own noncommercial "As-Is" license. FSLView seems to have its own separate GPL license: http://fsl.fmrib.ox.ac.uk/fsldownloads/ http://fsl.fmrib.ox.ac.uk/fsldownloads/ * AFNI used to require (free) registration, but is now GPL: https://afni.nimh.nih.gov/afni/about/legal https://afni.nimh.nih.gov/afni/about/legal * The BIC-MNI tools + source appear to be freely available for free under an "as-is" + attribution license. I only checked a few; please let me know if any are not and I will nag people accordingly: https://github.com/BIC-MNI https://github.com/BIC-MNI * Caret is GPL: http://brainvis.wustl.edu/wiki/index.php/Caret:About http://brainvis.wustl.edu/wiki/index.php/Caret:About * FreeSurfer has its own license which is very open: http://www.freesurfer.net/fswiki/FreeSurferSoftwareLicense http://www.freesurfer.net/fswiki/FreeSurferSoftwareLicense I suspect you're thinking of this, which reported differences between FreeSurfer versions: http://journals.plos.org/plosone/article/comment?id=info%3Adoi%2F10.1371%2Fannotation%2F2054bfcc-614f-4fe0-b1e4-72b6e89ba8e5 http://journals.plos.org/plosone/article/comment?id=info%3Ad... That turned out to be a combination of 1) algorithmic changes 2) Different random seeds (some of these functions use Monte Carlo-like methods), and 3) 32 vs 64 bit platforms.

verytrivial 10y ago

Citation?

op00to 10y ago

It's not that the research is 100% wrong, but the way the research is reported in the news is generally wrong. No, researchers can't really tell if you're thinking about a sailboat from looking at an fMRI, despite whatever the newspaper says. Source: worked in an fMRI research lab for 4 years.

acveilleux 10y ago

There's a fair bit of junky data and quite a few analysis methods that cannot compare data accurately between magnets, position wrt isocentre, or even processing software revisions. But your claim is a lot bolder than that.

Alex3917 10y ago
> There's a fair bit of junky data and quite a few analysis methods that cannot compare data accurately between magnets, position wrt isocentre, or even processing software revisions. But your claim is a lot bolder than that. What is the prevalence of each of those types of errors? Since AFAIK many of those errors occur in 30%+ of papers, unless you're assuming close to 0% independence, my claim doesn't seem especially bold...
- acveilleux 10y ago
  The methods I am familiar with, mostly dealing with volumetric measurements, registration, and automated segmentation were either robust or had well characterized limitations. I take some offense to blanketing the entire field of MRI based on a couple articles pointing out that some (admittedly a lot) fMRI experiments are statistically unsound.
  - Alex3917 10y ago
    How would you estimate the prevalence of inaccurate research in the field, if not by multiplying each (known) source of error by its estimated prevalence, and also using some estimate of the independence between errors? (And of course including general sources of error that affect all scientific research and aren't specific to any particular field.) And as far as independence goes, I haven't seen any research to suggest these errors have anything less than 100% independence, although if there is any research suggesting that these errors (or any other errors in science research) are correlated then I'd love to see that.

mattkrause 10y ago

There's this weird snobbishness about fMRI: it's uniquely terrible, the people are hacks, etc. It seems particularly common amongst first and second-year grad students who are doing something they think is "harder" science. I hate it. In the right hands, fMRI can be a really powerful technique for probing neural activity in healthy human subjects; in fact, it's one of the only ways to do so that has decent spatial resolution and thus lets you link brain structure and function. It certainly does have problems. There are plenty of ways to subtly mess up the data analysis or over-interpret results. The experimental design is often lacking, etc. However, I think these largely reflect the very low barrier of entry to fMRI research--all you really need is a laptop and somewhere willing to sell you scanner time (almost all major universities or hospitals)--rather than some intrinsic limitation of the field. The good work remains very good.