5 ms·

I think we are largely in agreement on the surface. The problem is that they are attributing the "positive" results to statistical error, when it isn't due to t

by nonbel 10y ago

I think we are largely in agreement on the surface. The problem is that they are attributing the "positive" results to statistical error, when it isn't due to that. The stats are working just fine and correctly telling us the null hypothesis is false. The problem is with the crappy null hypothesis, not the stats.

mattkrause 10y ago
The experiment is set up so that there are no true positives. When the threshold is p<0.05, then 5% of the calls should be positives, and they are all false positives. If anything else happens, SPM (etc) are borked. The only other alternatives are that the (fake) event design is somehow flawed in a way that exploits the non-stationarity of the fake event design, or the analysis code for the paper is somehow buggy. Are you trying to say that it's not the t-test per se, but something before it in the pipeline? If so, sure, of course, but that's not the null. The null is that chunks randomly labeled as ON are the same as chunks randomly labeled as OFF, and since it's not physically possible for the labels to affect the signal, this null must be true.
- nonbel 10y ago
  >"The experiment is set up so that there are no true positives." No, a true positive is when the null model is called false when it is false. They say the null is known to be false. The errors here are actually the false negatives, in this case that is every "non-significant" result. >"When the threshold is p<0.05, then 5% of the calls should be positives" Only if the null model is correct. In this case they describe problematic aspects of their null model. Ie they know it is false.
  - mattkrause 10y ago
    > They say the null is known to be false. Can you quote where they say that, because the article says: "Because two groups of subjects are randomly drawn from a large group of healthy controls, the null hypothesis of no group difference in brain activation should be true. Moreover, because the resting-state fMRI data should contain no consistent shifts in blood oxygen level-dependent (BOLD) activity, for a single group of subjects the null hypothesis of mean zero activation should also be true." There is also this quote: "Resting-state data should not contain systematic changes in brain activity, but our previous work (14) showed that the assumed activity paradigm can have a large impact on the degree of false positives. " This does not mean the null is false. That papers shows that temporal autocorrelations can also introduce spurious false positives. These factors affect block, but not event-related designs. They can be avoided by filtering--and maybe by averaging over many subjects (since these oscillations aren't locked to anything). In any case, they don't appear, as you can verify on the right side of Figure 1A. Finally, the discussion has a paragraph describing reasons why the experiment might be flawed, but most of those possible objections seem answered.
    - nonbel 10y ago
      The key point is that I am saying the second quote does say the null is false. This is where we differ. The actual null hypothesis they are testing is different than the one they actually care and talk about. It includes additional auxilliary assumptions that they say are wrong. What gets plugged in as the null hypothesis is not a problem in the realm of statistics. That is a choice of the researcher. The stats algorithm worked fine.
      - mattkrause 10y ago
        That sentence is closer to a link than actual content. If you look at paper (14), it has the following graph: http://imgur.com/a/oxSqp http://imgur.com/a/oxSqp It shows that the null model is perfectly appropriate for individual voxels under the event-related paradigms. The FWER there is indistinguishable from 5%, as it should be if the null were true. It also shows that FWER for the null model is inflated by ~6x for blocked paradigms. As they say in the new paper, it is unclear if that would re-occur in a multi-subject experiment (that was apparently a criticism of the first paper). Based on this, the absolute worst you can say is "the data on blocked paradigms is not very interesting", but that leaves their event-related data completely intact. As for the stats thing, the actual formal itself test is obvious fine--it's a t-test! They're saying that the pipeline that runs between the raw DICOM output of the scanner and the t-test, which is an equally important part of the testing procedure, is flawed. I'm happy to keep discussing this with you, but we seem to be going in circles. If "additional auxiliary assumptions" are the problem here, please describe them. If you think the 2012 NeuroImage paper proves that their null model is wrong in all cases, I'd love to read a few sentences describing how that follows.
        nonbel 10y ago
        >"They're saying that the pipeline that runs between the raw DICOM output of the scanner and the t-test, which is an equally important part of the testing procedure, is flawed." Yes, in other words the null hypothesis is being rendered false, by some aspect of the analysis pipeline. I am saying that is the correct description of the problem. Describing the problem as excess false positives is confused, because these are true positives. They appear to have successfully identified a problem, but then described and analyzed it incorrectly. >"The FWER there is indistinguishable from 5%, as it should be if the null were true." If the null is true, the pvalues should be samples from a uniform distribution. Another thing is they should have shown histograms of these. The 5% below 0.05 is not the whole story. There could be other ways the deviation manifests and/or they just chose sample sizes to be powered to get that result. "I'd love to read a few sentences describing how that follows." We have already read the exact sentences that mean the null model is false, multiple times. You have agreed that something was wrong, but want to call it something other than the null model.

by nonbel 10y ago

mattkrause 10y ago

The experiment is set up so that there are no true positives. When the threshold is p<0.05, then 5% of the calls should be positives, and they are all false positives. If anything else happens, SPM (etc) are borked. The only other alternatives are that the (fake) event design is somehow flawed in a way that exploits the non-stationarity of the fake event design, or the analysis code for the paper is somehow buggy. Are you trying to say that it's not the t-test per se, but something before it in the pipeline? If so, sure, of course, but that's not the null. The null is that chunks randomly labeled as ON are the same as chunks randomly labeled as OFF, and since it's not physically possible for the labels to affect the signal, this null must be true.

nonbel 10y ago
>"The experiment is set up so that there are no true positives." No, a true positive is when the null model is called false when it is false. They say the null is known to be false. The errors here are actually the false negatives, in this case that is every "non-significant" result. >"When the threshold is p<0.05, then 5% of the calls should be positives" Only if the null model is correct. In this case they describe problematic aspects of their null model. Ie they know it is false.
- mattkrause 10y ago
  > They say the null is known to be false. Can you quote where they say that, because the article says: "Because two groups of subjects are randomly drawn from a large group of healthy controls, the null hypothesis of no group difference in brain activation should be true. Moreover, because the resting-state fMRI data should contain no consistent shifts in blood oxygen level-dependent (BOLD) activity, for a single group of subjects the null hypothesis of mean zero activation should also be true." There is also this quote: "Resting-state data should not contain systematic changes in brain activity, but our previous work (14) showed that the assumed activity paradigm can have a large impact on the degree of false positives. " This does not mean the null is false. That papers shows that temporal autocorrelations can also introduce spurious false positives. These factors affect block, but not event-related designs. They can be avoided by filtering--and maybe by averaging over many subjects (since these oscillations aren't locked to anything). In any case, they don't appear, as you can verify on the right side of Figure 1A. Finally, the discussion has a paragraph describing reasons why the experiment might be flawed, but most of those possible objections seem answered.
  - nonbel 10y ago
    The key point is that I am saying the second quote does say the null is false. This is where we differ. The actual null hypothesis they are testing is different than the one they actually care and talk about. It includes additional auxilliary assumptions that they say are wrong. What gets plugged in as the null hypothesis is not a problem in the realm of statistics. That is a choice of the researcher. The stats algorithm worked fine.
    - mattkrause 10y ago
      That sentence is closer to a link than actual content. If you look at paper (14), it has the following graph: http://imgur.com/a/oxSqp http://imgur.com/a/oxSqp It shows that the null model is perfectly appropriate for individual voxels under the event-related paradigms. The FWER there is indistinguishable from 5%, as it should be if the null were true. It also shows that FWER for the null model is inflated by ~6x for blocked paradigms. As they say in the new paper, it is unclear if that would re-occur in a multi-subject experiment (that was apparently a criticism of the first paper). Based on this, the absolute worst you can say is "the data on blocked paradigms is not very interesting", but that leaves their event-related data completely intact. As for the stats thing, the actual formal itself test is obvious fine--it's a t-test! They're saying that the pipeline that runs between the raw DICOM output of the scanner and the t-test, which is an equally important part of the testing procedure, is flawed. I'm happy to keep discussing this with you, but we seem to be going in circles. If "additional auxiliary assumptions" are the problem here, please describe them. If you think the 2012 NeuroImage paper proves that their null model is wrong in all cases, I'd love to read a few sentences describing how that follows.
      - nonbel 10y ago
        >"They're saying that the pipeline that runs between the raw DICOM output of the scanner and the t-test, which is an equally important part of the testing procedure, is flawed." Yes, in other words the null hypothesis is being rendered false, by some aspect of the analysis pipeline. I am saying that is the correct description of the problem. Describing the problem as excess false positives is confused, because these are true positives. They appear to have successfully identified a problem, but then described and analyzed it incorrectly. >"The FWER there is indistinguishable from 5%, as it should be if the null were true." If the null is true, the pvalues should be samples from a uniform distribution. Another thing is they should have shown histograms of these. The 5% below 0.05 is not the whole story. There could be other ways the deviation manifests and/or they just chose sample sizes to be powered to get that result. "I'd love to read a few sentences describing how that follows." We have already read the exact sentences that mean the null model is false, multiple times. You have agreed that something was wrong, but want to call it something other than the null model.