16 ms·

fMRI software bugs could upend years of research

400 points by taylorbuley 10y ago

iamleppert 10y ago
Just goes to show when you're doing science you need to test and validate your experimental methodology, including the tools you use. In computer vision, its common to need to do some kind of calibration for many algorithms which can usually reveal some kind of statistical error or problem. I wonder why none of the researchers thought to do some very simple validation of the data? And I wonder if the software was at one point correct and then this bug was introduced at a later point? Many times it feels like after a company does a formal scientific validation they never do it again despite the fact they have engineers constantly working on the code...
chrramirez 10y ago
If this results to be true, this could be one of the most expensive bugs in computer history.
- FrojoS 10y ago
  Only, if you assume that those studies were of value without the error. obligatory xkcd: https://xkcd.com/1453/ https://xkcd.com/1453/
greenyoda 10y ago
Link to original paper: http://www.pnas.org/content/early/2016/06/27/1602413113.full http://www.pnas.org/content/early/2016/06/27/1602413113.full
williamscales 10y ago
"Our results suggest that the principal cause of the invalid cluster inferences is spatial autocorrelation functions that do not follow the assumed Gaussian shape." In other words, researchers cut corners. You should never assume that something is a certain way without rigorously proving it. How did these papers make it past peer review?
- Vexs 10y ago
  They were probably peer-reviewed by biologists, not statisticians. Peer review is mostly just a good think and read for about a month, not a perfect analysis.
  - jessriedel 10y ago
    ...as long as by "read for about a month" you mean "sit on it for a month, then read for a few hours".
    - leemailll 10y ago
      haha, so true
- trentmb 10y ago
  n=30 ought to be enough for anybody
  - leemailll 10y ago
    n>=30 is not the same as gaussian distribution
    - trentmb 10y ago
      Correct.
  - psycr 10y ago
    Depends how big the effect size is. For putting a loaded revolver next to your head, and pulling the trigger, n=30 is plenty.
    - trentmb 10y ago
      Correct.
- acveilleux 10y ago
jey 10y ago
Title should really read "fMRI" instead of "MRI". The referenced journal article is titled "Cluster failure: Why fMRI inferences for spatial extent have inflated false-positive rates".
- sxg 10y ago
  The same tools (AFNI, SPM, and FSL) are used in a lot of other MR research protocols as well - not just fMRI.
  - acveilleux 10y ago
    They are but 3dClustSim (the culprit) is essentially used only for fMRI.
    - dewarrn1 10y ago
      This is right and wrong: AFNI's 3dClustStim did contain a specific bug (since corrected) and is used for fMRI analysis, but the problem that the authors identified is more general than that and encompasses all of the tested packages.
- _delirium 10y ago
  Fwiw there's a response to an earlier (preprint) version of that paper from some of the developers of the packages in question. Preprint version of the "Cluster failure" paper, from last year: http://arxiv.org/abs/1511.01863 http://arxiv.org/abs/1511.01863 Response: http://arxiv.org/abs/1606.08199 http://arxiv.org/abs/1606.08199
  - dewarrn1 10y ago
    Flaundin & Friston's response is interesting because it essentially endorses the findings of Eklund et al. (except for one element of the Eklund analysis that they suggest is a modest error). F&F believe that by setting one parameter correctly (i.e., using a conservative cluster forming threshold) the validity of their preferred parametric statistical approach is upheld. Eklund et al. might quibble because their take-home message is that non-parametric methods should be used instead, but their findings are not misrepresented by F&F. Regardless, an open and important question is how often other authors used a sufficiently conservative cluster forming threshold for their fMRI analyses. If nothing else, Eklund et al. will cause future reports to be more cautious in this regard.
Alex3917 10y ago
It's already been known for several years that almost all MRI brain scan research is wrong, what exactly is new here?
- JumpCrisscross 10y ago
  > almost all MRI brain scan research is wrong Source for a layman?
  - goalieca 10y ago
    I wasn't in fmri research but some fellow students in my lab were on it. I do know there was a study with dead salmon which were showing up as having "active" brain regions. I have to find it but am on my phone. It's easy to imagine the main difficulty with this is mapping signal to actual thoughts and regions. It's a really complex biology and physics to reduce.
    - leemailll 10y ago
      what is your field? Neuroscience?
      - goalieca 10y ago
        Electrical engineering (medical imaging - medical imaging technologies) This is the best I could find for the paper. not what I originally meant to cite. http://onlinelibrary.wiley.com/doi/10.1002/scin.5591761320/abstract http://onlinelibrary.wiley.com/doi/10.1002/scin.5591761320/a...
    - greenyoda 10y ago
      Searching for "dead salmon fMRI" turned up the following: http://www.prefrontal.org/files/posters/Bennett-Salmon-2009.pdf http://www.prefrontal.org/files/posters/Bennett-Salmon-2009....
      - acveilleux 10y ago
honkhonkpants 10y ago
Doesn't sound like a straight up bug, but rather unsound statistical methods which can happen with or without software. You get the same problem with finite element analysis software: the operator has to be aware of all the assumptions baked in, and has to ensure that the input conforms to them.
- greenyoda 10y ago
  I'd say it's a bug, since the unsound statistical methods are incorporated into the three most common software packages that are specifically intended to be used for fMRI analysis. If these methods don't work well for fMRI data, fMRI software shouldn't be using them. From the paper: "Using this null data with different experimental designs, we estimate the incidence of significant results. In theory, we should find 5% false positives (for a significance threshold of 5%), but instead we found that the most common software packages for fMRI analysis (SPM, FSL, AFNI) can result in false-positive rates of up to 70%." (http://www.pnas.org/content/early/2016/06/27/1602413113.full http://www.pnas.org/content/early/2016/06/27/1602413113.full) Choosing a bad algorithm for your software is as much of a bug as a null-pointer crash, and is something that needs to be tested for when building software.
  - sverige 10y ago
    And it's a little disheartening when it takes this long to find such a significant bug. Lots of work gone to waste. I wonder if the data can be rerun? Sounds like lots of the raw data wasn't archived correctly or at all.
    - nonbel 10y ago
      The real "bug" here isn't even really statistical. It is the usual logical issue that has been well known for years (yet the problem has only grown and spread): "In most psychological research, improved power of a statistical design leads to a prior probability approaching 1/2 of finding a significant difference in the theoretically predicted direction. Hence the corroboration yielded by “success” is very weak, and becomes weaker with increased precision. “Statistical significance” plays a logical role in psychology precisely the reverse of its role in physics." http://cerco.ups-tlse.fr/pdf0609/Meehl_1967.pdf http://cerco.ups-tlse.fr/pdf0609/Meehl_1967.pdf In other words, the fMRI researchers have been testing a null hypothesis known to be false beforehand, so statistical significance tells us nothing of value, and definitely nothing about the theory of interest. This paper is really interesting, because the authors don't seem to realize that this is what they are discovering. Getting a "statistically significant" result is the correct answer for the tests to return, these are not false positives. The problem is a poorly chosen null model.
iLoch 10y ago
"How X looks like" -- what's with this gramatical mistake? I see it everywhere. Is it a regional thing?
- strictfp 10y ago
  It's a common error for native Swedes, It's a direct translation from Swedish. Could be true for other languages as well, I believe most Germanic languages use "how".
  - waqf 10y ago
    English uses "how" too: it's correct to say "how X looks", with no "like". "what … like" is synonymous with "how".
Toenex 10y ago
And from a couple of days before. https://news.ycombinator.com/item?id=12019205 https://news.ycombinator.com/item?id=12019205
nonbel 10y ago
I would send it back and ask for a detailed description of the null hypothesis they are testing, because they are not clear on this point at all: >"All of the analyses to this point have been based on resting-state fMRI data, where the null hypothesis should be true." They are not careful to explicitly define this null hypothesis anywhere, but earlier in the paper they describe some issues with the model used: >"Resting-state data should not contain systematic changes in brain activity, but our previous work (14) showed that the assumed activity paradigm can have a large impact on the degree of false positives. Several different activity paradigms were therefore used, two block based (B1 and B2) and two event related (E1 and E2); see Table 1 for details." This means that they actually know the null model to be false and have even written papers about some of the major contributors to this: >"The main reason for the high familywise error rates seems to be that the global AR(1) auto correlation correction in SPM fails to model the spectra of the residuals" http://www.sciencedirect.com/science/article/pii/S1053811912003825 http://www.sciencedirect.com/science/article/pii/S1053811912... If the null hypothesis is false, it is no wonder they detect this. In fact, if the sample size was larger (they used only n=20/40 here) they would get near 100% false positive rates. The test seems to be telling them the truth, it is a trivial truth, but according to their description it is correct nonetheless. Edit: I was quoting from the actual paper. http://www.pnas.org/content/early/2016/06/27/1602413113.full http://www.pnas.org/content/early/2016/06/27/1602413113.full
- mattkrause 10y ago
  Nononono. The null hypothesis here is definitely true. They take resting state data ("lie here while we scan your brain") and pretend that it was acquired during an experiment, using either block or event-related designs. Next, they look for differences between the fake blocks (or events). The null hypothesis is that neural activity is the same across blocks. It must be true because the block boundaries are totally arbitrary. In fact, they were defined after the scanning, so they could not possibly influence neural activity (assuming the resting state data is stationary, which may or may not be true; they discuss that). The NeuroImage paper you linked to isn't disagreeing with this. They're saying that if you use the above procedure to make your null model, it doesn't work and it doesn't work because of the reason outlined in this paper.
randcraw 10y ago
As someone who works in the biomedical imaging business and is also a fan of philosophy, I think this news will matter more to folks in the latter camp. For a couple years now philosophers have insisted that fMRI images prove there is no such thing as free will. Today's revelation should put an end to that whole line of reasoning (and the absurd amount of fatalism that it engendered). (The back story: Apparently fMRI showed motor signals arising before the cognitive / conscious signals that should have created them, assuming we humans have free will. This has led to the widely adopted belief among philosophers that we humans act before we think, thus we don't and can't act willfully and freely. To wit, science has proven there is no such thing as free will; we're all just automatons.) Just this week there was an article in The Atlantic on how we all must accept that we're mere robots and we don't really choose our actions (nor can we choose to believe in a god). Ah well. It seems philosophers STILL haven't learned the importance of applying the scientific method before leaping to a conclusion -- sometimes just to check that someone else didn't just abuse the scientific method.
- tucosan 10y ago
  So have these studies been invalidated by the software bug as well? If so, do you have any pointers? I. e. which were the infamous studies, and did they indeed use the faulty software to derive their conclusions? I'm genuinely interested.
  - randcraw 10y ago
    http://www.theatlantic.com/magazine/archive/2016/06/theres-no-such-thing-as-free-will/480750/ http://www.theatlantic.com/magazine/archive/2016/06/theres-n... This article preceded awareness of the bug.
- nitrogen 10y ago
  There are loads of other reasons not to accept dualistic free will.
  - gmac 10y ago
    Yes: see e.g. https://www.rep.routledge.com/articles/free-will/v-1 https://www.rep.routledge.com/articles/free-will/v-1
Trombone12 10y ago
Well, I think the problems with interpreting fMRI scans have been at least vaguely known since that time a dead salmon activated its neurons when asked to judge the of a person from a photo, this was in 2009. http://www.wired.com/2009/09/fmrisalmon/ http://www.wired.com/2009/09/fmrisalmon/
- dewarrn1 10y ago
  The dead salmon article is a bit of a red herring here. It's a clear demonstration that a shoddy statistical approach can undermine fMRI findings. Critically, the implications of the current paper extend to even research that has been rigorously analyzed using field-standard software. Statistical issues are at the heart of both papers, but the newer paper identifies problems that are subtle and ubiquitous.
- omginternets 10y ago
  I don't see how that's really relevant. The point of the dead salmon paper was "do your damn FWE corrections". The point of paper, is rather "popular tools make unsound assumptions" (or, inversely, "researchers are violating the assumptions made by their tools"). The problem in the salmon paper is easy to fix. The problem in this paper is trickier.
jcoffland 10y ago
This is why I always eat my science which a large helping of humble pie with extra skepticism.
SubiculumCode 10y ago
What a crap article.
jballanc 10y ago
The real takeaway lesson from this research should be the vital importance of Open Data to the modern scientific enterprise: > "lamentable archiving and data-sharing practices" that prevent most of the discipline's body of work being re-analysed. Keeping data private before publication is (at this point in time) understandable. Once results are published, however, there is no excuse for not depositing the raw data in an open repository for later re-evaluation.
- toyg 10y ago
  Uh, where would you store it though? IIRC from the time a relative was going through a chemistry PHD, they produced several GBs of raw data every half hour. Storage is cheap but it's not that cheap...
  - pas 10y ago
    Even storing - and making accessible - a fraction of it would be a drastic improvement over the current status quo.
  - tansey 10y ago
    It's not that much data. You're storing about 2M voxel image every 2s or so. There are open repositories out there already: https://openfmri.org/ https://openfmri.org/
  - zachrose 10y ago
    But how expensive was the lab and the instruments that produce that amount of data?
    - jballanc 10y ago
      Not to mention the researchers' and post-docs' and students' time. By not paying to store it, you're risking the possibility that all of this money is essentially flushed down the toilet by a software bug because it cannot now be reanalyzed.
      - 10y ago
deleted 10y ago
[deleted]
deleted 10y ago
[deleted]
pfooti 10y ago
The dead salmon study seems relevant here in discussion of how fMRI is used, especially the theory -ladenness of observations. http://blogs.scientificamerican.com/scicurious-brain/ignobel-prize-in-neuroscience-the-dead-salmon-study/ http://blogs.scientificamerican.com/scicurious-brain/ignobel...
- deleted 10y ago
  [deleted]
- nkurz 10y ago
  Maybe I missed the link, but the full text of the readable, relevant, and enjoyable article that blog post discusses is here: The principled control of false positives in neuroimaging Bennett, Wolford, Miller 2009 http://scan.oxfordjournals.org/content/4/4/417.full http://scan.oxfordjournals.org/content/4/4/417.full
  - pfooti 10y ago
    Thanks. I'm on a mobile in a foreign land, so had issues tracking down a useful link.
multinglets 10y ago
So people don't perform complex, goal-focused motor tasks without having a goal ahead of time after all. Wow, philosophy people. EDIT: Cry about it all you want. It won't change the fact that in 100 years people will look back and wonder if an entire academic discipline was afflicted with some form of literal mental retardation.
jamesrom 10y ago
I thought that fMRI hasn't been taken seriously since they scanned a dead fish and found it was thinking.
- remriel 10y ago
  That was just a false positive.
- omginternets 10y ago
  >fMRI hasn't been taken seriously I don't know where you got that (very strange) impression. The dead fish paper showed that multiple comparisons yield false positives, which is a problem that is much broader than fMRI methodology. With proper method, fMRI is a very reliable and insightful tool. It's just very difficult to do properly.
nerdponx 10y ago
"Further: “Our results suggest that the principal cause of the invalid cluster inferences is spatial autocorrelation functions that do not follow the assumed Gaussian shape”." This has nothing to do with bugs and everything to do with bad statistical analysis. It's Google Flu all over again.
- AlexCoventry 10y ago
  It's all relative. At the right level of abstraction, bad statistical analysis is a bug.
seesomesense 10y ago
fMRI is great for generating headlines and pretty pictures for the popular media. However most neurologists view the vast majority of fMRI research as junk science.
bjourne 10y ago
Does it mean studies like these are likely bunk? http://kangleelab.com/articles/Paper0002_0009.pdf https://med.stanford.edu/news/all-news/2016/05/moms-voice-activates-different-regions-in-children-brains.html https://www.theguardian.com/science/2015/apr/21/babies-feel-pain-like-adults-mri-scan-study-suggests https://news.brown.edu/articles/2013/06/breastfeeding And with bunk I mean doesn't show what they claim.
maweki 10y ago
"and along the way they swipe the fMRI community for their “lamentable archiving and data-sharing practices” that prevent most of the discipline's body of work being re-analysed." That's quite funny. My girlfriend recently finished her master's thesis on data sharing for neuroscience data and created a model for universal access to research data across institutions, but came to the conclusion that making researchers share their data is a bigger hurdle than actually implementing the system. The main reason for lack of sharing, she postulated, is, that studies (that create funding for the researcher who publishes them) can be done using just the raw data and researchers who create data want to publish all the studies/papers for themselves (because "they" paid for the data acquisition) and are also afraid to publish underlying data for it to be harder for others to falsify their results, which would lead, in their opinion, to funding going away. Edit: of course there are privacy issues for the test subjects as well.
- kriro 10y ago
  Please ask her to keep working on the problem and keep fighting the good fight. I think it's a political problem. If research grants were only given out if all data was made available (and while we're at it open source all the code produced) we'd be a huge step further down the right path. There is very, very strong resistance to these sort of ideas. The idealist in me likes to think that there shouldn't be much resistance after all scientists should be happy if their results get double checked (for free!). I can only assume that a lot of people are very afraid of having their data double checked (or possibly never had data in the first place). Additionally access to data is a competitive advantage as it's often hard to generate.
- brandonl222 10y ago
  It's a rather sad reflection of society, no telling how may breakthroughs we're missing out on due to the lack of cooperation in science on this front.
  - lostlogin 10y ago
    There is also the motivations of the scanner owner to consider. They are paid to do scans. A cheap scan might be low hundreds of dollars per hour. A more commercial rate would be well into the thousands. We don't want to dry up now do we?
UVDMAS 10y ago
The paper has been rebutted by other researchers who argue that the original results hold: "This technical report revisits the analysis of family-wise error rates in statistical parametric mapping - using random field theory - reported in (Eklund et al., 2015). Contrary to the understandable spin that these sorts of analyses attract, a review of their results suggests that they endorse the use of parametric assumptions - and random field theory - in the analysis of functional neuroimaging data. We briefly rehearse the advantages parametric analyses offer over nonparametric alternatives and then unpack the implications of (Eklund et al., 2015) for parametric procedures." http://arxiv.org/abs/1606.08199 http://arxiv.org/abs/1606.08199
- dewarrn1 10y ago
  Sort of. The rebuttal (by Flandin and Friston) suggests that properly-applied parametric statistics of the kind they favor are valid. Eklund et al. wouldn't disagree with that because their own findings support it, but they would point out that not all researchers necessarily adhered to the conservative statistical approach that F&F discuss. More specifically, both sets of authors describe the importance of using a conservative "cluster defining threshold" to identify spatially contiguous 3D blobs of brain activation. Eklund et al. use their findings to raise the question of whether the bulk of fMRI reports were conservative in this regard.

400 points by taylorbuley 10y ago

Just goes to show when you're doing science you need to test and validate your experimental methodology, including the tools you use. In computer vision, its common to need to do some kind of calibration for many algorithms which can usually reveal some kind of statistical error or problem. I wonder why none of the researchers thought to do some very simple validation of the data? And I wonder if the software was at one point correct and then this bug was introduced at a later point? Many times it feels like after a company does a formal scientific validation they never do it again despite the fact they have engineers constantly working on the code...

chrramirez 10y ago

If this results to be true, this could be one of the most expensive bugs in computer history.

FrojoS 10y ago
Only, if you assume that those studies were of value without the error. obligatory xkcd: https://xkcd.com/1453/ https://xkcd.com/1453/

greenyoda 10y ago

Link to original paper: http://www.pnas.org/content/early/2016/06/27/1602413113.full http://www.pnas.org/content/early/2016/06/27/1602413113.full

williamscales 10y ago

"Our results suggest that the principal cause of the invalid cluster inferences is spatial autocorrelation functions that do not follow the assumed Gaussian shape." In other words, researchers cut corners. You should never assume that something is a certain way without rigorously proving it. How did these papers make it past peer review?

Vexs 10y ago
They were probably peer-reviewed by biologists, not statisticians. Peer review is mostly just a good think and read for about a month, not a perfect analysis.
- jessriedel 10y ago
  ...as long as by "read for about a month" you mean "sit on it for a month, then read for a few hours".
  - leemailll 10y ago
    haha, so true
trentmb 10y ago
n=30 ought to be enough for anybody
- leemailll 10y ago
  n>=30 is not the same as gaussian distribution
  - trentmb 10y ago
    Correct.
- psycr 10y ago
  Depends how big the effect size is. For putting a loaded revolver next to your head, and pulling the trigger, n=30 is plenty.
  - trentmb 10y ago
    Correct.
acveilleux 10y ago

jey 10y ago

Title should really read "fMRI" instead of "MRI". The referenced journal article is titled "Cluster failure: Why fMRI inferences for spatial extent have inflated false-positive rates".

sxg 10y ago
The same tools (AFNI, SPM, and FSL) are used in a lot of other MR research protocols as well - not just fMRI.
- acveilleux 10y ago
  They are but 3dClustSim (the culprit) is essentially used only for fMRI.
  - dewarrn1 10y ago
    This is right and wrong: AFNI's 3dClustStim did contain a specific bug (since corrected) and is used for fMRI analysis, but the problem that the authors identified is more general than that and encompasses all of the tested packages.
_delirium 10y ago
Fwiw there's a response to an earlier (preprint) version of that paper from some of the developers of the packages in question. Preprint version of the "Cluster failure" paper, from last year: http://arxiv.org/abs/1511.01863 http://arxiv.org/abs/1511.01863 Response: http://arxiv.org/abs/1606.08199 http://arxiv.org/abs/1606.08199
- dewarrn1 10y ago
  Flaundin & Friston's response is interesting because it essentially endorses the findings of Eklund et al. (except for one element of the Eklund analysis that they suggest is a modest error). F&F believe that by setting one parameter correctly (i.e., using a conservative cluster forming threshold) the validity of their preferred parametric statistical approach is upheld. Eklund et al. might quibble because their take-home message is that non-parametric methods should be used instead, but their findings are not misrepresented by F&F. Regardless, an open and important question is how often other authors used a sufficiently conservative cluster forming threshold for their fMRI analyses. If nothing else, Eklund et al. will cause future reports to be more cautious in this regard.

Alex3917 10y ago

It's already been known for several years that almost all MRI brain scan research is wrong, what exactly is new here?

JumpCrisscross 10y ago
> almost all MRI brain scan research is wrong Source for a layman?
- goalieca 10y ago
  I wasn't in fmri research but some fellow students in my lab were on it. I do know there was a study with dead salmon which were showing up as having "active" brain regions. I have to find it but am on my phone. It's easy to imagine the main difficulty with this is mapping signal to actual thoughts and regions. It's a really complex biology and physics to reduce.
  - leemailll 10y ago
    what is your field? Neuroscience?
    - goalieca 10y ago
      Electrical engineering (medical imaging - medical imaging technologies) This is the best I could find for the paper. not what I originally meant to cite. http://onlinelibrary.wiley.com/doi/10.1002/scin.5591761320/abstract http://onlinelibrary.wiley.com/doi/10.1002/scin.5591761320/a...
  - greenyoda 10y ago
    Searching for "dead salmon fMRI" turned up the following: http://www.prefrontal.org/files/posters/Bennett-Salmon-2009.pdf http://www.prefrontal.org/files/posters/Bennett-Salmon-2009....
    - acveilleux 10y ago

honkhonkpants 10y ago

Doesn't sound like a straight up bug, but rather unsound statistical methods which can happen with or without software. You get the same problem with finite element analysis software: the operator has to be aware of all the assumptions baked in, and has to ensure that the input conforms to them.

greenyoda 10y ago
I'd say it's a bug, since the unsound statistical methods are incorporated into the three most common software packages that are specifically intended to be used for fMRI analysis. If these methods don't work well for fMRI data, fMRI software shouldn't be using them. From the paper: "Using this null data with different experimental designs, we estimate the incidence of significant results. In theory, we should find 5% false positives (for a significance threshold of 5%), but instead we found that the most common software packages for fMRI analysis (SPM, FSL, AFNI) can result in false-positive rates of up to 70%." (http://www.pnas.org/content/early/2016/06/27/1602413113.full http://www.pnas.org/content/early/2016/06/27/1602413113.full) Choosing a bad algorithm for your software is as much of a bug as a null-pointer crash, and is something that needs to be tested for when building software.
- sverige 10y ago
  And it's a little disheartening when it takes this long to find such a significant bug. Lots of work gone to waste. I wonder if the data can be rerun? Sounds like lots of the raw data wasn't archived correctly or at all.
  - nonbel 10y ago
    The real "bug" here isn't even really statistical. It is the usual logical issue that has been well known for years (yet the problem has only grown and spread): "In most psychological research, improved power of a statistical design leads to a prior probability approaching 1/2 of finding a significant difference in the theoretically predicted direction. Hence the corroboration yielded by “success” is very weak, and becomes weaker with increased precision. “Statistical significance” plays a logical role in psychology precisely the reverse of its role in physics." http://cerco.ups-tlse.fr/pdf0609/Meehl_1967.pdf http://cerco.ups-tlse.fr/pdf0609/Meehl_1967.pdf In other words, the fMRI researchers have been testing a null hypothesis known to be false beforehand, so statistical significance tells us nothing of value, and definitely nothing about the theory of interest. This paper is really interesting, because the authors don't seem to realize that this is what they are discovering. Getting a "statistically significant" result is the correct answer for the tests to return, these are not false positives. The problem is a poorly chosen null model.

iLoch 10y ago

"How X looks like" -- what's with this gramatical mistake? I see it everywhere. Is it a regional thing?

strictfp 10y ago
It's a common error for native Swedes, It's a direct translation from Swedish. Could be true for other languages as well, I believe most Germanic languages use "how".
- waqf 10y ago
  English uses "how" too: it's correct to say "how X looks", with no "like". "what … like" is synonymous with "how".

Toenex 10y ago

And from a couple of days before. https://news.ycombinator.com/item?id=12019205 https://news.ycombinator.com/item?id=12019205

nonbel 10y ago

I would send it back and ask for a detailed description of the null hypothesis they are testing, because they are not clear on this point at all: >"All of the analyses to this point have been based on resting-state fMRI data, where the null hypothesis should be true." They are not careful to explicitly define this null hypothesis anywhere, but earlier in the paper they describe some issues with the model used: >"Resting-state data should not contain systematic changes in brain activity, but our previous work (14) showed that the assumed activity paradigm can have a large impact on the degree of false positives. Several different activity paradigms were therefore used, two block based (B1 and B2) and two event related (E1 and E2); see Table 1 for details." This means that they actually know the null model to be false and have even written papers about some of the major contributors to this: >"The main reason for the high familywise error rates seems to be that the global AR(1) auto correlation correction in SPM fails to model the spectra of the residuals" http://www.sciencedirect.com/science/article/pii/S1053811912003825 http://www.sciencedirect.com/science/article/pii/S1053811912... If the null hypothesis is false, it is no wonder they detect this. In fact, if the sample size was larger (they used only n=20/40 here) they would get near 100% false positive rates. The test seems to be telling them the truth, it is a trivial truth, but according to their description it is correct nonetheless. Edit: I was quoting from the actual paper. http://www.pnas.org/content/early/2016/06/27/1602413113.full http://www.pnas.org/content/early/2016/06/27/1602413113.full

mattkrause 10y ago
Nononono. The null hypothesis here is definitely true. They take resting state data ("lie here while we scan your brain") and pretend that it was acquired during an experiment, using either block or event-related designs. Next, they look for differences between the fake blocks (or events). The null hypothesis is that neural activity is the same across blocks. It must be true because the block boundaries are totally arbitrary. In fact, they were defined after the scanning, so they could not possibly influence neural activity (assuming the resting state data is stationary, which may or may not be true; they discuss that). The NeuroImage paper you linked to isn't disagreeing with this. They're saying that if you use the above procedure to make your null model, it doesn't work and it doesn't work because of the reason outlined in this paper.

randcraw 10y ago

As someone who works in the biomedical imaging business and is also a fan of philosophy, I think this news will matter more to folks in the latter camp. For a couple years now philosophers have insisted that fMRI images prove there is no such thing as free will. Today's revelation should put an end to that whole line of reasoning (and the absurd amount of fatalism that it engendered). (The back story: Apparently fMRI showed motor signals arising before the cognitive / conscious signals that should have created them, assuming we humans have free will. This has led to the widely adopted belief among philosophers that we humans act before we think, thus we don't and can't act willfully and freely. To wit, science has proven there is no such thing as free will; we're all just automatons.) Just this week there was an article in The Atlantic on how we all must accept that we're mere robots and we don't really choose our actions (nor can we choose to believe in a god). Ah well. It seems philosophers STILL haven't learned the importance of applying the scientific method before leaping to a conclusion -- sometimes just to check that someone else didn't just abuse the scientific method.

tucosan 10y ago
So have these studies been invalidated by the software bug as well? If so, do you have any pointers? I. e. which were the infamous studies, and did they indeed use the faulty software to derive their conclusions? I'm genuinely interested.
- randcraw 10y ago
  http://www.theatlantic.com/magazine/archive/2016/06/theres-no-such-thing-as-free-will/480750/ http://www.theatlantic.com/magazine/archive/2016/06/theres-n... This article preceded awareness of the bug.
nitrogen 10y ago
There are loads of other reasons not to accept dualistic free will.
- gmac 10y ago
  Yes: see e.g. https://www.rep.routledge.com/articles/free-will/v-1 https://www.rep.routledge.com/articles/free-will/v-1

Trombone12 10y ago

Well, I think the problems with interpreting fMRI scans have been at least vaguely known since that time a dead salmon activated its neurons when asked to judge the of a person from a photo, this was in 2009. http://www.wired.com/2009/09/fmrisalmon/ http://www.wired.com/2009/09/fmrisalmon/

dewarrn1 10y ago
The dead salmon article is a bit of a red herring here. It's a clear demonstration that a shoddy statistical approach can undermine fMRI findings. Critically, the implications of the current paper extend to even research that has been rigorously analyzed using field-standard software. Statistical issues are at the heart of both papers, but the newer paper identifies problems that are subtle and ubiquitous.
omginternets 10y ago
I don't see how that's really relevant. The point of the dead salmon paper was "do your damn FWE corrections". The point of paper, is rather "popular tools make unsound assumptions" (or, inversely, "researchers are violating the assumptions made by their tools"). The problem in the salmon paper is easy to fix. The problem in this paper is trickier.

jcoffland 10y ago

This is why I always eat my science which a large helping of humble pie with extra skepticism.

SubiculumCode 10y ago

What a crap article.

jballanc 10y ago

The real takeaway lesson from this research should be the vital importance of Open Data to the modern scientific enterprise: > "lamentable archiving and data-sharing practices" that prevent most of the discipline's body of work being re-analysed. Keeping data private before publication is (at this point in time) understandable. Once results are published, however, there is no excuse for not depositing the raw data in an open repository for later re-evaluation.

toyg 10y ago
Uh, where would you store it though? IIRC from the time a relative was going through a chemistry PHD, they produced several GBs of raw data every half hour. Storage is cheap but it's not that cheap...
- pas 10y ago
  Even storing - and making accessible - a fraction of it would be a drastic improvement over the current status quo.
- tansey 10y ago
  It's not that much data. You're storing about 2M voxel image every 2s or so. There are open repositories out there already: https://openfmri.org/ https://openfmri.org/
- zachrose 10y ago
  But how expensive was the lab and the instruments that produce that amount of data?
  - jballanc 10y ago
    Not to mention the researchers' and post-docs' and students' time. By not paying to store it, you're risking the possibility that all of this money is essentially flushed down the toilet by a software bug because it cannot now be reanalyzed.
    - 10y ago

deleted 10y ago

[deleted]

deleted 10y ago

[deleted]

pfooti 10y ago

The dead salmon study seems relevant here in discussion of how fMRI is used, especially the theory -ladenness of observations. http://blogs.scientificamerican.com/scicurious-brain/ignobel-prize-in-neuroscience-the-dead-salmon-study/ http://blogs.scientificamerican.com/scicurious-brain/ignobel...

deleted 10y ago
[deleted]
nkurz 10y ago
Maybe I missed the link, but the full text of the readable, relevant, and enjoyable article that blog post discusses is here: The principled control of false positives in neuroimaging Bennett, Wolford, Miller 2009 http://scan.oxfordjournals.org/content/4/4/417.full http://scan.oxfordjournals.org/content/4/4/417.full
- pfooti 10y ago
  Thanks. I'm on a mobile in a foreign land, so had issues tracking down a useful link.

multinglets 10y ago

So people don't perform complex, goal-focused motor tasks without having a goal ahead of time after all. Wow, philosophy people. EDIT: Cry about it all you want. It won't change the fact that in 100 years people will look back and wonder if an entire academic discipline was afflicted with some form of literal mental retardation.

jamesrom 10y ago

I thought that fMRI hasn't been taken seriously since they scanned a dead fish and found it was thinking.

remriel 10y ago
That was just a false positive.
omginternets 10y ago
>fMRI hasn't been taken seriously I don't know where you got that (very strange) impression. The dead fish paper showed that multiple comparisons yield false positives, which is a problem that is much broader than fMRI methodology. With proper method, fMRI is a very reliable and insightful tool. It's just very difficult to do properly.

nerdponx 10y ago

"Further: “Our results suggest that the principal cause of the invalid cluster inferences is spatial autocorrelation functions that do not follow the assumed Gaussian shape”." This has nothing to do with bugs and everything to do with bad statistical analysis. It's Google Flu all over again.

AlexCoventry 10y ago
It's all relative. At the right level of abstraction, bad statistical analysis is a bug.

seesomesense 10y ago

fMRI is great for generating headlines and pretty pictures for the popular media. However most neurologists view the vast majority of fMRI research as junk science.

bjourne 10y ago

Does it mean studies like these are likely bunk? http://kangleelab.com/articles/Paper0002_0009.pdf https://med.stanford.edu/news/all-news/2016/05/moms-voice-activates-different-regions-in-children-brains.html https://www.theguardian.com/science/2015/apr/21/babies-feel-pain-like-adults-mri-scan-study-suggests https://news.brown.edu/articles/2013/06/breastfeeding And with bunk I mean doesn't show what they claim.

maweki 10y ago

"and along the way they swipe the fMRI community for their “lamentable archiving and data-sharing practices” that prevent most of the discipline's body of work being re-analysed." That's quite funny. My girlfriend recently finished her master's thesis on data sharing for neuroscience data and created a model for universal access to research data across institutions, but came to the conclusion that making researchers share their data is a bigger hurdle than actually implementing the system. The main reason for lack of sharing, she postulated, is, that studies (that create funding for the researcher who publishes them) can be done using just the raw data and researchers who create data want to publish all the studies/papers for themselves (because "they" paid for the data acquisition) and are also afraid to publish underlying data for it to be harder for others to falsify their results, which would lead, in their opinion, to funding going away. Edit: of course there are privacy issues for the test subjects as well.

kriro 10y ago
Please ask her to keep working on the problem and keep fighting the good fight. I think it's a political problem. If research grants were only given out if all data was made available (and while we're at it open source all the code produced) we'd be a huge step further down the right path. There is very, very strong resistance to these sort of ideas. The idealist in me likes to think that there shouldn't be much resistance after all scientists should be happy if their results get double checked (for free!). I can only assume that a lot of people are very afraid of having their data double checked (or possibly never had data in the first place). Additionally access to data is a competitive advantage as it's often hard to generate.
brandonl222 10y ago
It's a rather sad reflection of society, no telling how may breakthroughs we're missing out on due to the lack of cooperation in science on this front.
- lostlogin 10y ago
  There is also the motivations of the scanner owner to consider. They are paid to do scans. A cheap scan might be low hundreds of dollars per hour. A more commercial rate would be well into the thousands. We don't want to dry up now do we?

UVDMAS 10y ago

The paper has been rebutted by other researchers who argue that the original results hold: "This technical report revisits the analysis of family-wise error rates in statistical parametric mapping - using random field theory - reported in (Eklund et al., 2015). Contrary to the understandable spin that these sorts of analyses attract, a review of their results suggests that they endorse the use of parametric assumptions - and random field theory - in the analysis of functional neuroimaging data. We briefly rehearse the advantages parametric analyses offer over nonparametric alternatives and then unpack the implications of (Eklund et al., 2015) for parametric procedures." http://arxiv.org/abs/1606.08199 http://arxiv.org/abs/1606.08199

dewarrn1 10y ago
Sort of. The rebuttal (by Flandin and Friston) suggests that properly-applied parametric statistics of the kind they favor are valid. Eklund et al. wouldn't disagree with that because their own findings support it, but they would point out that not all researchers necessarily adhered to the conservative statistical approach that F&F discuss. More specifically, both sets of authors describe the importance of using a conservative "cluster defining threshold" to identify spatially contiguous 3D blobs of brain activation. Eklund et al. use their findings to raise the question of whether the bulk of fMRI reports were conservative in this regard.