5 ms·

>"Can you explain what those assumptions are then, even roughly?" I have not looked at the code myself. I only read their description of it. As I quoted earlie

by nonbel 10y ago

>"Can you explain what those assumptions are then, even roughly?" I have not looked at the code myself. I only read their description of it. As I quoted earlier: "assumed activity paradigm can have a large impact on the degree of false positives" If the assumed activity paradigm affects (what they are calling) the false positive rate, it is obviously part of the model. If they are arbitrarily varying it because they don't know which one is "best", it is clearly wrong. The t-tests will pick up this wrong assumption if there is enough data to do so (even without knowing the details, remember they say it affects the "false positive" rate). The t-test is doing it's job correctly, however, they are plugging in a known to be incorrect null hypothesis, rendering this testing procedure pointless.

mattkrause 10y ago
Maybe we're agreeing? The ENTIRE POINT of the paper is that there are inappropriate assumptions baked into the defaults of FSL and SPM. (AFNI has an actual code-does-the-wrong-thing bug). These errors result in cluster-wise tests being inappropriately sized: when thresholds are set to p<0.001, there are more than 0.1% false positives. They tested this by dividing a single stream of "off" data into segments of "ON" and "OFF" data. They do this in four different ways (two block paradigms and two event-related paradigms). Most fMRI experiments use a paradigm that is close to one of these, so this isn't a fishing attempt--it's meant to make the results broadly applicable . They argue that, by construction, no ON data is present in this pseudo-data. Therefore, any ON calls must be spurious/false alarms/type I errors/etc, which lets you calculate the actual FWER and compare it with its nominal size. You seem to be arguing that this step is not true--some other assumption, which they knew about(?), causes it. The NeuroImage paper says that different assumed activity paradigms have different false positive rates because the assumed model is dumb. The event-related paradigms alternate between on and off rapidly (and at random), which washes out the effects of unmodelled low frequencies. The much slower block designs, on the other hand, are especially susceptible to this problem because they spend a long time in each condition. The current work extends that to look at other ways in which the default assumptions might be dumb in SPM, while adding in more data, different tests, and other packages. I am open to the argument that the Block Design data (e.g., in Figure 2 in PNAS) does not add much because the NeuroImage paper already suggests those designs are screwed. However, it also says that the event-related designs are actually in pretty good shape: the voxel-based familywise error rates in the NeuroImage paper are about right (Figure 2), but the cluster FWER for the same design is through the roof in the PNAS paper. Otherwise though, I don't see many unwarranted/undiscussed assumptions in the paper. If you want to keep claiming that the authors (and I) are making wild assumptions, it would be nice to have a vague sense of what they are and why they/I am wrong.
- nonbel 10y ago
  I think we are largely in agreement on the surface. The problem is that they are attributing the "positive" results to statistical error, when it isn't due to that. The stats are working just fine and correctly telling us the null hypothesis is false. The problem is with the crappy null hypothesis, not the stats.
  - mattkrause 10y ago
    The experiment is set up so that there are no true positives. When the threshold is p<0.05, then 5% of the calls should be positives, and they are all false positives. If anything else happens, SPM (etc) are borked. The only other alternatives are that the (fake) event design is somehow flawed in a way that exploits the non-stationarity of the fake event design, or the analysis code for the paper is somehow buggy. Are you trying to say that it's not the t-test per se, but something before it in the pipeline? If so, sure, of course, but that's not the null. The null is that chunks randomly labeled as ON are the same as chunks randomly labeled as OFF, and since it's not physically possible for the labels to affect the signal, this null must be true.
    - nonbel 10y ago
      >"The experiment is set up so that there are no true positives." No, a true positive is when the null model is called false when it is false. They say the null is known to be false. The errors here are actually the false negatives, in this case that is every "non-significant" result. >"When the threshold is p<0.05, then 5% of the calls should be positives" Only if the null model is correct. In this case they describe problematic aspects of their null model. Ie they know it is false.
      - mattkrause 10y ago
        > They say the null is known to be false. Can you quote where they say that, because the article says: "Because two groups of subjects are randomly drawn from a large group of healthy controls, the null hypothesis of no group difference in brain activation should be true. Moreover, because the resting-state fMRI data should contain no consistent shifts in blood oxygen level-dependent (BOLD) activity, for a single group of subjects the null hypothesis of mean zero activation should also be true." There is also this quote: "Resting-state data should not contain systematic changes in brain activity, but our previous work (14) showed that the assumed activity paradigm can have a large impact on the degree of false positives. " This does not mean the null is false. That papers shows that temporal autocorrelations can also introduce spurious false positives. These factors affect block, but not event-related designs. They can be avoided by filtering--and maybe by averaging over many subjects (since these oscillations aren't locked to anything). In any case, they don't appear, as you can verify on the right side of Figure 1A. Finally, the discussion has a paragraph describing reasons why the experiment might be flawed, but most of those possible objections seem answered.

by nonbel 10y ago

mattkrause 10y ago

Maybe we're agreeing? The ENTIRE POINT of the paper is that there are inappropriate assumptions baked into the defaults of FSL and SPM. (AFNI has an actual code-does-the-wrong-thing bug). These errors result in cluster-wise tests being inappropriately sized: when thresholds are set to p<0.001, there are more than 0.1% false positives. They tested this by dividing a single stream of "off" data into segments of "ON" and "OFF" data. They do this in four different ways (two block paradigms and two event-related paradigms). Most fMRI experiments use a paradigm that is close to one of these, so this isn't a fishing attempt--it's meant to make the results broadly applicable . They argue that, by construction, no ON data is present in this pseudo-data. Therefore, any ON calls must be spurious/false alarms/type I errors/etc, which lets you calculate the actual FWER and compare it with its nominal size. You seem to be arguing that this step is not true--some other assumption, which they knew about(?), causes it. The NeuroImage paper says that different assumed activity paradigms have different false positive rates because the assumed model is dumb. The event-related paradigms alternate between on and off rapidly (and at random), which washes out the effects of unmodelled low frequencies. The much slower block designs, on the other hand, are especially susceptible to this problem because they spend a long time in each condition. The current work extends that to look at other ways in which the default assumptions might be dumb in SPM, while adding in more data, different tests, and other packages. I am open to the argument that the Block Design data (e.g., in Figure 2 in PNAS) does not add much because the NeuroImage paper already suggests those designs are screwed. However, it also says that the event-related designs are actually in pretty good shape: the voxel-based familywise error rates in the NeuroImage paper are about right (Figure 2), but the cluster FWER for the same design is through the roof in the PNAS paper. Otherwise though, I don't see many unwarranted/undiscussed assumptions in the paper. If you want to keep claiming that the authors (and I) are making wild assumptions, it would be nice to have a vague sense of what they are and why they/I am wrong.

nonbel 10y ago
I think we are largely in agreement on the surface. The problem is that they are attributing the "positive" results to statistical error, when it isn't due to that. The stats are working just fine and correctly telling us the null hypothesis is false. The problem is with the crappy null hypothesis, not the stats.
- mattkrause 10y ago
  The experiment is set up so that there are no true positives. When the threshold is p<0.05, then 5% of the calls should be positives, and they are all false positives. If anything else happens, SPM (etc) are borked. The only other alternatives are that the (fake) event design is somehow flawed in a way that exploits the non-stationarity of the fake event design, or the analysis code for the paper is somehow buggy. Are you trying to say that it's not the t-test per se, but something before it in the pipeline? If so, sure, of course, but that's not the null. The null is that chunks randomly labeled as ON are the same as chunks randomly labeled as OFF, and since it's not physically possible for the labels to affect the signal, this null must be true.
  - nonbel 10y ago
    >"The experiment is set up so that there are no true positives." No, a true positive is when the null model is called false when it is false. They say the null is known to be false. The errors here are actually the false negatives, in this case that is every "non-significant" result. >"When the threshold is p<0.05, then 5% of the calls should be positives" Only if the null model is correct. In this case they describe problematic aspects of their null model. Ie they know it is false.
    - mattkrause 10y ago
      > They say the null is known to be false. Can you quote where they say that, because the article says: "Because two groups of subjects are randomly drawn from a large group of healthy controls, the null hypothesis of no group difference in brain activation should be true. Moreover, because the resting-state fMRI data should contain no consistent shifts in blood oxygen level-dependent (BOLD) activity, for a single group of subjects the null hypothesis of mean zero activation should also be true." There is also this quote: "Resting-state data should not contain systematic changes in brain activity, but our previous work (14) showed that the assumed activity paradigm can have a large impact on the degree of false positives. " This does not mean the null is false. That papers shows that temporal autocorrelations can also introduce spurious false positives. These factors affect block, but not event-related designs. They can be avoided by filtering--and maybe by averaging over many subjects (since these oscillations aren't locked to anything). In any case, they don't appear, as you can verify on the right side of Figure 1A. Finally, the discussion has a paragraph describing reasons why the experiment might be flawed, but most of those possible objections seem answered.