3 ms·

The key point is that I am saying the second quote does say the null is false. This is where we differ. The actual null hypothesis they are testing is different

by nonbel 10y ago

The key point is that I am saying the second quote does say the null is false. This is where we differ. The actual null hypothesis they are testing is different than the one they actually care and talk about. It includes additional auxilliary assumptions that they say are wrong. What gets plugged in as the null hypothesis is not a problem in the realm of statistics. That is a choice of the researcher. The stats algorithm worked fine.

mattkrause 10y ago
That sentence is closer to a link than actual content. If you look at paper (14), it has the following graph: http://imgur.com/a/oxSqp http://imgur.com/a/oxSqp It shows that the null model is perfectly appropriate for individual voxels under the event-related paradigms. The FWER there is indistinguishable from 5%, as it should be if the null were true. It also shows that FWER for the null model is inflated by ~6x for blocked paradigms. As they say in the new paper, it is unclear if that would re-occur in a multi-subject experiment (that was apparently a criticism of the first paper). Based on this, the absolute worst you can say is "the data on blocked paradigms is not very interesting", but that leaves their event-related data completely intact. As for the stats thing, the actual formal itself test is obvious fine--it's a t-test! They're saying that the pipeline that runs between the raw DICOM output of the scanner and the t-test, which is an equally important part of the testing procedure, is flawed. I'm happy to keep discussing this with you, but we seem to be going in circles. If "additional auxiliary assumptions" are the problem here, please describe them. If you think the 2012 NeuroImage paper proves that their null model is wrong in all cases, I'd love to read a few sentences describing how that follows.
- nonbel 10y ago
  >"They're saying that the pipeline that runs between the raw DICOM output of the scanner and the t-test, which is an equally important part of the testing procedure, is flawed." Yes, in other words the null hypothesis is being rendered false, by some aspect of the analysis pipeline. I am saying that is the correct description of the problem. Describing the problem as excess false positives is confused, because these are true positives. They appear to have successfully identified a problem, but then described and analyzed it incorrectly. >"The FWER there is indistinguishable from 5%, as it should be if the null were true." If the null is true, the pvalues should be samples from a uniform distribution. Another thing is they should have shown histograms of these. The 5% below 0.05 is not the whole story. There could be other ways the deviation manifests and/or they just chose sample sizes to be powered to get that result. "I'd love to read a few sentences describing how that follows." We have already read the exact sentences that mean the null model is false, multiple times. You have agreed that something was wrong, but want to call it something other than the null model.
  - deleted 10y ago
    [deleted]
  - ihnorton 10y ago
    > Describing the problem as excess false positives is confused, because these are true positives. This is deeply wrong. A "true positive" in this context would mean that the resting brain activity of multiple subjects is actually correlated with arbitrary length, randomized, moving test windows. Again, the data is untreated; they are imposing arbitrary test windows and looking for increases in brain activity (increased image intensity) correlated with the stimulus-ON sections of the pseudo-treatment. From reading some of your prior comments on the subject, it seems prudent to point out that the causal link between stimulus and BOLD effect is quite literally observable: http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3130346/ http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3130346/
    - nonbel 10y ago
      >"This is deeply wrong. A "true positive" in this context would mean that the resting brain activity of multiple subjects is actually correlated with arbitrary length, randomized, moving test windows." This is the same misconception I have been trying to dispel. The null hypothesis is not the inverse of that "layman's" statement, it is a specific set of predicted results calculated in a very specific way. It is a mathematical statement, not an English prose statement. In this case, apparently one part of this calculation (this assumption about autocorrelation, whatever it is doesn't matter to my point) has lead to such a large deviation from the observations that the null model has been rejected. The null model has been rejected correctly. This is not a false positive. The problem here is not false-positives due to faulty stats. It is the poor mapping between the hypothesis the researchers want to test, and the null hypothesis they are actually testing. The tools provided by statisticians look like they worked just fine in this case. If the researchers have decided to use them for GIGO, that is not a statistical problem.
      - ihnorton 10y ago
        > This is the same misconception I have been trying to dispel. The null hypothesis is not the inverse of that "layman's" statement, it is a specific set of predicted results calculated in a very specific way. The entire point of this discussion is that the software is not calculating the null hypothesis that users expect. That the current model it does in fact calculate is internally-consistent is tautological and irrelevant (though actual bugs were found in at least one package). As you yourself said: "I didn't read the code, or even the paper very closely." (https://news.ycombinator.com/item?id=12037207 https://news.ycombinator.com/item?id=12037207) Perhaps you should do?
- nonbel 10y ago
  Maybe this will help. I am thinking of the same distinction discussed by Meehl in Figure 2 of this paper: "The poor social scientist, confronted with the twofold problem of dangerous inferential passage (right-to-left) in Figure 2 is rescued as to the (H → O) problem by the statistician. Comforted by these “objective” inferential tools (formulas and tables), the social scientist easily forgets about the far more serious, and less tractable, (T → H) problem, which the statistics text does not address." http://rhowell.ba.ttu.edu/Meehl1.pdf http://rhowell.ba.ttu.edu/Meehl1.pdf Here, we are not even yet talking about the substantive theory, although that issue will also exist. There is also a difference between the statistical hypothesis (the mathematical object) and what the researcher wants to test. I think you are conflating those two hypotheses.

by nonbel 10y ago

mattkrause 10y ago

That sentence is closer to a link than actual content. If you look at paper (14), it has the following graph: http://imgur.com/a/oxSqp http://imgur.com/a/oxSqp It shows that the null model is perfectly appropriate for individual voxels under the event-related paradigms. The FWER there is indistinguishable from 5%, as it should be if the null were true. It also shows that FWER for the null model is inflated by ~6x for blocked paradigms. As they say in the new paper, it is unclear if that would re-occur in a multi-subject experiment (that was apparently a criticism of the first paper). Based on this, the absolute worst you can say is "the data on blocked paradigms is not very interesting", but that leaves their event-related data completely intact. As for the stats thing, the actual formal itself test is obvious fine--it's a t-test! They're saying that the pipeline that runs between the raw DICOM output of the scanner and the t-test, which is an equally important part of the testing procedure, is flawed. I'm happy to keep discussing this with you, but we seem to be going in circles. If "additional auxiliary assumptions" are the problem here, please describe them. If you think the 2012 NeuroImage paper proves that their null model is wrong in all cases, I'd love to read a few sentences describing how that follows.

nonbel 10y ago
>"They're saying that the pipeline that runs between the raw DICOM output of the scanner and the t-test, which is an equally important part of the testing procedure, is flawed." Yes, in other words the null hypothesis is being rendered false, by some aspect of the analysis pipeline. I am saying that is the correct description of the problem. Describing the problem as excess false positives is confused, because these are true positives. They appear to have successfully identified a problem, but then described and analyzed it incorrectly. >"The FWER there is indistinguishable from 5%, as it should be if the null were true." If the null is true, the pvalues should be samples from a uniform distribution. Another thing is they should have shown histograms of these. The 5% below 0.05 is not the whole story. There could be other ways the deviation manifests and/or they just chose sample sizes to be powered to get that result. "I'd love to read a few sentences describing how that follows." We have already read the exact sentences that mean the null model is false, multiple times. You have agreed that something was wrong, but want to call it something other than the null model.
- deleted 10y ago
  [deleted]
- ihnorton 10y ago
  > Describing the problem as excess false positives is confused, because these are true positives. This is deeply wrong. A "true positive" in this context would mean that the resting brain activity of multiple subjects is actually correlated with arbitrary length, randomized, moving test windows. Again, the data is untreated; they are imposing arbitrary test windows and looking for increases in brain activity (increased image intensity) correlated with the stimulus-ON sections of the pseudo-treatment. From reading some of your prior comments on the subject, it seems prudent to point out that the causal link between stimulus and BOLD effect is quite literally observable: http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3130346/ http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3130346/
  - nonbel 10y ago
    >"This is deeply wrong. A "true positive" in this context would mean that the resting brain activity of multiple subjects is actually correlated with arbitrary length, randomized, moving test windows." This is the same misconception I have been trying to dispel. The null hypothesis is not the inverse of that "layman's" statement, it is a specific set of predicted results calculated in a very specific way. It is a mathematical statement, not an English prose statement. In this case, apparently one part of this calculation (this assumption about autocorrelation, whatever it is doesn't matter to my point) has lead to such a large deviation from the observations that the null model has been rejected. The null model has been rejected correctly. This is not a false positive. The problem here is not false-positives due to faulty stats. It is the poor mapping between the hypothesis the researchers want to test, and the null hypothesis they are actually testing. The tools provided by statisticians look like they worked just fine in this case. If the researchers have decided to use them for GIGO, that is not a statistical problem.
    - ihnorton 10y ago
      > This is the same misconception I have been trying to dispel. The null hypothesis is not the inverse of that "layman's" statement, it is a specific set of predicted results calculated in a very specific way. The entire point of this discussion is that the software is not calculating the null hypothesis that users expect. That the current model it does in fact calculate is internally-consistent is tautological and irrelevant (though actual bugs were found in at least one package). As you yourself said: "I didn't read the code, or even the paper very closely." (https://news.ycombinator.com/item?id=12037207 https://news.ycombinator.com/item?id=12037207) Perhaps you should do?
nonbel 10y ago
Maybe this will help. I am thinking of the same distinction discussed by Meehl in Figure 2 of this paper: "The poor social scientist, confronted with the twofold problem of dangerous inferential passage (right-to-left) in Figure 2 is rescued as to the (H → O) problem by the statistician. Comforted by these “objective” inferential tools (formulas and tables), the social scientist easily forgets about the far more serious, and less tractable, (T → H) problem, which the statistics text does not address." http://rhowell.ba.ttu.edu/Meehl1.pdf http://rhowell.ba.ttu.edu/Meehl1.pdf Here, we are not even yet talking about the substantive theory, although that issue will also exist. There is also a difference between the statistical hypothesis (the mathematical object) and what the researcher wants to test. I think you are conflating those two hypotheses.