5 ms·

Doesn't sound like a straight up bug, but rather unsound statistical methods which can happen with or without software. You get the same problem with finite ele

Doesn't sound like a straight up bug, but rather unsound statistical methods which can happen with or without software. You get the same problem with finite element analysis software: the operator has to be aware of all the assumptions baked in, and has to ensure that the input conforms to them.

greenyoda 10y ago
I'd say it's a bug, since the unsound statistical methods are incorporated into the three most common software packages that are specifically intended to be used for fMRI analysis. If these methods don't work well for fMRI data, fMRI software shouldn't be using them. From the paper: "Using this null data with different experimental designs, we estimate the incidence of significant results. In theory, we should find 5% false positives (for a significance threshold of 5%), but instead we found that the most common software packages for fMRI analysis (SPM, FSL, AFNI) can result in false-positive rates of up to 70%." (http://www.pnas.org/content/early/2016/06/27/1602413113.full http://www.pnas.org/content/early/2016/06/27/1602413113.full) Choosing a bad algorithm for your software is as much of a bug as a null-pointer crash, and is something that needs to be tested for when building software.
- sverige 10y ago
  And it's a little disheartening when it takes this long to find such a significant bug. Lots of work gone to waste. I wonder if the data can be rerun? Sounds like lots of the raw data wasn't archived correctly or at all.
  - nonbel 10y ago
    The real "bug" here isn't even really statistical. It is the usual logical issue that has been well known for years (yet the problem has only grown and spread): "In most psychological research, improved power of a statistical design leads to a prior probability approaching 1/2 of finding a significant difference in the theoretically predicted direction. Hence the corroboration yielded by “success” is very weak, and becomes weaker with increased precision. “Statistical significance” plays a logical role in psychology precisely the reverse of its role in physics." http://cerco.ups-tlse.fr/pdf0609/Meehl_1967.pdf http://cerco.ups-tlse.fr/pdf0609/Meehl_1967.pdf In other words, the fMRI researchers have been testing a null hypothesis known to be false beforehand, so statistical significance tells us nothing of value, and definitely nothing about the theory of interest. This paper is really interesting, because the authors don't seem to realize that this is what they are discovering. Getting a "statistically significant" result is the correct answer for the tests to return, these are not false positives. The problem is a poorly chosen null model.
    - mattkrause 10y ago
      Are you sure about that? As I understand it, they are testing a really-really-null hypothesis. Specifically, they started with some resting state data (a condition where the subjects just lie there, attempting to remain alive). They then took this data and imposed fake task structures over it, as if the subject were doing two different things (e.g., condition #1 starts at t=0, t=1 min, t=3 min, t=6 min; condition #2 starts at t=2, t=4, t=5 min). Once this is done, the task+data is submitted to their analysis pipeline. The null hypothesis (fake condition #1 == fake condition #2) has to be true, by construction. This assumes that the original data doesn't have any structure, which may or may not be true (they discuss it a bit towards the end of the paper).
      - nonbel 10y ago
        >"The null hypothesis (fake condition #1 == fake condition #2) has to be true, by construction." The null hypothesis is not only that condition 1 == condition 2, it involves other assumptions being made. From their own description, they knew it was false before even doing this analysis and used this knowledge to design the study. Apparently there is some incorrect assumption about autocorrelation being made. It seems the authors did not really grasp what hypothesis they were testing, leading to their confused (but still productive) description of the problem. I go into more in this post: https://news.ycombinator.com/item?id=12032772 https://news.ycombinator.com/item?id=12032772
- thaw13579 10y ago
  No, a statistical modeling assumption is not the same as a software bug. There is no perfect model, and nonparametric tests are approximations at best. The important thing is that the publication reports the modeling assumption so that readers can judge the conclusions relative to that choice. Now we know to judge results using the Gaussian model to have a high false positive rate. On the other hand, no researcher would knowingly publish results affected by software bugs, e.g. null pointers or arithmetic errors... hopefully.
- moptar 10y ago
  All finite element analysis software uses "bad algorithms" in the sense that they give wrong results if the user doesn't take their limitations into account. A common type of error causes them to underestimate stress at the highest stress locations - where failure is most likely! Other problems occur with structures that stiffen as they're loaded. That effect is ignored by linear analysis and engineers often don't do the more accurate non-linear analysis because it cost more time (computer resources) and money (more expensive licenses) as well as sometimes just being more difficult to set up. It's more that the technology isn't advanced enough to be foolproof than that there are bugs.
dewarrn1 10y ago
You're correct that the main finding reflects flawed statistical assumptions made by the major neuroimaging packages (or so these authors contend), but they did uncover a specific bug in one of the three packages (AFNI): "... a 15-year-old bug was found in 3dClustSim while testing the three software packages (the bug was fixed by the AFNI group as of May 2015, during preparation of this manuscript). The bug essentially reduced the size of the image searched for clusters, underestimating the severity of the multiplicity correction and overestimating significance ..."

greenyoda 10y ago

I'd say it's a bug, since the unsound statistical methods are incorporated into the three most common software packages that are specifically intended to be used for fMRI analysis. If these methods don't work well for fMRI data, fMRI software shouldn't be using them. From the paper: "Using this null data with different experimental designs, we estimate the incidence of significant results. In theory, we should find 5% false positives (for a significance threshold of 5%), but instead we found that the most common software packages for fMRI analysis (SPM, FSL, AFNI) can result in false-positive rates of up to 70%." (http://www.pnas.org/content/early/2016/06/27/1602413113.full http://www.pnas.org/content/early/2016/06/27/1602413113.full) Choosing a bad algorithm for your software is as much of a bug as a null-pointer crash, and is something that needs to be tested for when building software.

sverige 10y ago
And it's a little disheartening when it takes this long to find such a significant bug. Lots of work gone to waste. I wonder if the data can be rerun? Sounds like lots of the raw data wasn't archived correctly or at all.
- nonbel 10y ago
  The real "bug" here isn't even really statistical. It is the usual logical issue that has been well known for years (yet the problem has only grown and spread): "In most psychological research, improved power of a statistical design leads to a prior probability approaching 1/2 of finding a significant difference in the theoretically predicted direction. Hence the corroboration yielded by “success” is very weak, and becomes weaker with increased precision. “Statistical significance” plays a logical role in psychology precisely the reverse of its role in physics." http://cerco.ups-tlse.fr/pdf0609/Meehl_1967.pdf http://cerco.ups-tlse.fr/pdf0609/Meehl_1967.pdf In other words, the fMRI researchers have been testing a null hypothesis known to be false beforehand, so statistical significance tells us nothing of value, and definitely nothing about the theory of interest. This paper is really interesting, because the authors don't seem to realize that this is what they are discovering. Getting a "statistically significant" result is the correct answer for the tests to return, these are not false positives. The problem is a poorly chosen null model.
  - mattkrause 10y ago
    Are you sure about that? As I understand it, they are testing a really-really-null hypothesis. Specifically, they started with some resting state data (a condition where the subjects just lie there, attempting to remain alive). They then took this data and imposed fake task structures over it, as if the subject were doing two different things (e.g., condition #1 starts at t=0, t=1 min, t=3 min, t=6 min; condition #2 starts at t=2, t=4, t=5 min). Once this is done, the task+data is submitted to their analysis pipeline. The null hypothesis (fake condition #1 == fake condition #2) has to be true, by construction. This assumes that the original data doesn't have any structure, which may or may not be true (they discuss it a bit towards the end of the paper).
    - nonbel 10y ago
      >"The null hypothesis (fake condition #1 == fake condition #2) has to be true, by construction." The null hypothesis is not only that condition 1 == condition 2, it involves other assumptions being made. From their own description, they knew it was false before even doing this analysis and used this knowledge to design the study. Apparently there is some incorrect assumption about autocorrelation being made. It seems the authors did not really grasp what hypothesis they were testing, leading to their confused (but still productive) description of the problem. I go into more in this post: https://news.ycombinator.com/item?id=12032772 https://news.ycombinator.com/item?id=12032772
thaw13579 10y ago
No, a statistical modeling assumption is not the same as a software bug. There is no perfect model, and nonparametric tests are approximations at best. The important thing is that the publication reports the modeling assumption so that readers can judge the conclusions relative to that choice. Now we know to judge results using the Gaussian model to have a high false positive rate. On the other hand, no researcher would knowingly publish results affected by software bugs, e.g. null pointers or arithmetic errors... hopefully.
moptar 10y ago
All finite element analysis software uses "bad algorithms" in the sense that they give wrong results if the user doesn't take their limitations into account. A common type of error causes them to underestimate stress at the highest stress locations - where failure is most likely! Other problems occur with structures that stiffen as they're loaded. That effect is ignored by linear analysis and engineers often don't do the more accurate non-linear analysis because it cost more time (computer resources) and money (more expensive licenses) as well as sometimes just being more difficult to set up. It's more that the technology isn't advanced enough to be foolproof than that there are bugs.

dewarrn1 10y ago

You're correct that the main finding reflects flawed statistical assumptions made by the major neuroimaging packages (or so these authors contend), but they did uncover a specific bug in one of the three packages (AFNI): "... a 15-year-old bug was found in 3dClustSim while testing the three software packages (the bug was fixed by the AFNI group as of May 2015, during preparation of this manuscript). The bug essentially reduced the size of the image searched for clusters, underestimating the severity of the multiplicity correction and overestimating significance ..."