5 ms·

I would send it back and ask for a detailed description of the null hypothesis they are testing, because they are not clear on this point at all: >"All of the

by nonbel 10y ago

I would send it back and ask for a detailed description of the null hypothesis they are testing, because they are not clear on this point at all: >"All of the analyses to this point have been based on resting-state fMRI data, where the null hypothesis should be true." They are not careful to explicitly define this null hypothesis anywhere, but earlier in the paper they describe some issues with the model used: >"Resting-state data should not contain systematic changes in brain activity, but our previous work (14) showed that the assumed activity paradigm can have a large impact on the degree of false positives. Several different activity paradigms were therefore used, two block based (B1 and B2) and two event related (E1 and E2); see Table 1 for details." This means that they actually know the null model to be false and have even written papers about some of the major contributors to this: >"The main reason for the high familywise error rates seems to be that the global AR(1) auto correlation correction in SPM fails to model the spectra of the residuals" http://www.sciencedirect.com/science/article/pii/S1053811912003825 http://www.sciencedirect.com/science/article/pii/S1053811912... If the null hypothesis is false, it is no wonder they detect this. In fact, if the sample size was larger (they used only n=20/40 here) they would get near 100% false positive rates. The test seems to be telling them the truth, it is a trivial truth, but according to their description it is correct nonetheless. Edit: I was quoting from the actual paper. http://www.pnas.org/content/early/2016/06/27/1602413113.full http://www.pnas.org/content/early/2016/06/27/1602413113.full

mattkrause 10y ago
Nononono. The null hypothesis here is definitely true. They take resting state data ("lie here while we scan your brain") and pretend that it was acquired during an experiment, using either block or event-related designs. Next, they look for differences between the fake blocks (or events). The null hypothesis is that neural activity is the same across blocks. It must be true because the block boundaries are totally arbitrary. In fact, they were defined after the scanning, so they could not possibly influence neural activity (assuming the resting state data is stationary, which may or may not be true; they discuss that). The NeuroImage paper you linked to isn't disagreeing with this. They're saying that if you use the above procedure to make your null model, it doesn't work and it doesn't work because of the reason outlined in this paper.
- nonbel 10y ago
  >"The null hypothesis is that neural activity is the same across blocks." You may have written this before seeing my other response, but I will repeat the point here. You have mentioned only one part of the null hypothesis, there are other assumptions being made that can be violated. From their description, it sounds like they discovered one that is causing the null model to make bad predictions. Btw, I am using "null hypothesis" and "null model "interchangeably here. Really, null model should be the preferred term but people are so used to "hypothesis".
  - mattkrause 10y ago
    Can you explain what those assumptions are then, even roughly? I read the PNAS paper as being an extension of their previous work, not a contradiction of it. They've scaled the data set up (the previous paper had less data), added more packages (the previous paper used only SPM), and ran it as a group analysis (the previous paper was within subjects). They pretty much say that right in the PNAS text: "That work found a high degree of false positives, up to 70% compared with the expected 5%, likely due to a simplistic temporal autocorrelation model in SPM. It was, however, not clear whether these problems would propagate to group studies. Another unanswered question was the statistical validity of other fMRI software packages. We address these limitations in the current work with an evaluation of group inference with the three most common fMRI software packages [SPM (15, 16), FSL (17), and AFNI (18)]." Furthermore, the NeuroImage paper says that the AR(1) temporal autocorrelation model in SPM is actually not too problematic for event-related designs: "Overall, a simple AR(1) model for temporal correlations appears to be adequate for fast designs (E1 and E2) at all three TRs. However, there is a massive inflation of false positive rates at short TRs that is particularly pronounced for slower (block) designs. " If that's what is bugging you, just ignore the red and yellow bars in the PNAS figures.
    - nonbel 10y ago
      >"Can you explain what those assumptions are then, even roughly?" I have not looked at the code myself. I only read their description of it. As I quoted earlier: "assumed activity paradigm can have a large impact on the degree of false positives" If the assumed activity paradigm affects (what they are calling) the false positive rate, it is obviously part of the model. If they are arbitrarily varying it because they don't know which one is "best", it is clearly wrong. The t-tests will pick up this wrong assumption if there is enough data to do so (even without knowing the details, remember they say it affects the "false positive" rate). The t-test is doing it's job correctly, however, they are plugging in a known to be incorrect null hypothesis, rendering this testing procedure pointless.
nickledave 10y ago
See mattkrause's comment below. I think you might not be understanding what they're testing. They're taking only resting state data, randomly sorting some of it into "pretend active state data" and then asking whether they get any statistically significant difference between these two groups when they shouldn't. But they do. That means the tests they used, the same tests many authors use, are giving false positives. The null hypothesis is "there will be no difference between testing state data and whatever data we get when we ask the subject to do some activity". They can "reject" that hypothesis using only randomly shuffled resting state data, so there's something wrong with the stats packages themselves
- nickledave 10y ago
  @mattkrause: you beat me to it
  - mattkrause 10y ago
    Eh, you'll scoop me on something when it matters. PS: If you are the song bird guy, I saw a talk of yours on youtube. Cool stuff.
- nonbel 10y ago
  I think that like mattkrause (and the authors of the current work), you have forgotten that the null hypothesis is something larger than condition 1 == condition 2. There are various other components, usually (somewhat misleadingly) referred to as assumptions, that can also cause the predictions derived from the null model to deviate from the data. For stuff like a t-test, one parameter value (ie the mean of the distribution) gets all the attention. But this is wrong, it is only one part of the model being tested.
  - nickledave 10y ago
    I'm aware that there are assumptions implicit in what the null hypothesis is. You are the one who keeps saying the authors don't even realize what those assumptions are, but you haven't pointed out anything besides what the authors said. What are the other faulty assumptions you've identified that the authors are missing? I guess you're referring to some sort of issue with power that you mentioned in your previous comment?

by nonbel 10y ago

mattkrause 10y ago

Nononono. The null hypothesis here is definitely true. They take resting state data ("lie here while we scan your brain") and pretend that it was acquired during an experiment, using either block or event-related designs. Next, they look for differences between the fake blocks (or events). The null hypothesis is that neural activity is the same across blocks. It must be true because the block boundaries are totally arbitrary. In fact, they were defined after the scanning, so they could not possibly influence neural activity (assuming the resting state data is stationary, which may or may not be true; they discuss that). The NeuroImage paper you linked to isn't disagreeing with this. They're saying that if you use the above procedure to make your null model, it doesn't work and it doesn't work because of the reason outlined in this paper.

nonbel 10y ago
>"The null hypothesis is that neural activity is the same across blocks." You may have written this before seeing my other response, but I will repeat the point here. You have mentioned only one part of the null hypothesis, there are other assumptions being made that can be violated. From their description, it sounds like they discovered one that is causing the null model to make bad predictions. Btw, I am using "null hypothesis" and "null model "interchangeably here. Really, null model should be the preferred term but people are so used to "hypothesis".
- mattkrause 10y ago
  Can you explain what those assumptions are then, even roughly? I read the PNAS paper as being an extension of their previous work, not a contradiction of it. They've scaled the data set up (the previous paper had less data), added more packages (the previous paper used only SPM), and ran it as a group analysis (the previous paper was within subjects). They pretty much say that right in the PNAS text: "That work found a high degree of false positives, up to 70% compared with the expected 5%, likely due to a simplistic temporal autocorrelation model in SPM. It was, however, not clear whether these problems would propagate to group studies. Another unanswered question was the statistical validity of other fMRI software packages. We address these limitations in the current work with an evaluation of group inference with the three most common fMRI software packages [SPM (15, 16), FSL (17), and AFNI (18)]." Furthermore, the NeuroImage paper says that the AR(1) temporal autocorrelation model in SPM is actually not too problematic for event-related designs: "Overall, a simple AR(1) model for temporal correlations appears to be adequate for fast designs (E1 and E2) at all three TRs. However, there is a massive inflation of false positive rates at short TRs that is particularly pronounced for slower (block) designs. " If that's what is bugging you, just ignore the red and yellow bars in the PNAS figures.
  - nonbel 10y ago
    >"Can you explain what those assumptions are then, even roughly?" I have not looked at the code myself. I only read their description of it. As I quoted earlier: "assumed activity paradigm can have a large impact on the degree of false positives" If the assumed activity paradigm affects (what they are calling) the false positive rate, it is obviously part of the model. If they are arbitrarily varying it because they don't know which one is "best", it is clearly wrong. The t-tests will pick up this wrong assumption if there is enough data to do so (even without knowing the details, remember they say it affects the "false positive" rate). The t-test is doing it's job correctly, however, they are plugging in a known to be incorrect null hypothesis, rendering this testing procedure pointless.

nickledave 10y ago

See mattkrause's comment below. I think you might not be understanding what they're testing. They're taking only resting state data, randomly sorting some of it into "pretend active state data" and then asking whether they get any statistically significant difference between these two groups when they shouldn't. But they do. That means the tests they used, the same tests many authors use, are giving false positives. The null hypothesis is "there will be no difference between testing state data and whatever data we get when we ask the subject to do some activity". They can "reject" that hypothesis using only randomly shuffled resting state data, so there's something wrong with the stats packages themselves

nickledave 10y ago
@mattkrause: you beat me to it
- mattkrause 10y ago
  Eh, you'll scoop me on something when it matters. PS: If you are the song bird guy, I saw a talk of yours on youtube. Cool stuff.
nonbel 10y ago
I think that like mattkrause (and the authors of the current work), you have forgotten that the null hypothesis is something larger than condition 1 == condition 2. There are various other components, usually (somewhat misleadingly) referred to as assumptions, that can also cause the predictions derived from the null model to deviate from the data. For stuff like a t-test, one parameter value (ie the mean of the distribution) gets all the attention. But this is wrong, it is only one part of the model being tested.
- nickledave 10y ago
  I'm aware that there are assumptions implicit in what the null hypothesis is. You are the one who keeps saying the authors don't even realize what those assumptions are, but you haven't pointed out anything besides what the authors said. What are the other faulty assumptions you've identified that the authors are missing? I guess you're referring to some sort of issue with power that you mentioned in your previous comment?