5 ms·

They should change the nomenclature "significant", which in normal language tends to imply a large size to statistically "detectable" or "discernible". P=0.05

They should change the nomenclature "significant", which in normal language tends to imply a large size to statistically "detectable" or "discernible". P=0.05 is best interpreted as "There is a 95% chance that this data isn't pure noise". Then researchers should always put more emphasis on confidence intervals. Readers of papers can then see if the size is enough to make the results relevant and not likely caused by experimental accident. Plus given the perverse incentives researchers are subject to, maybe assume that the real effect size is probably closer to the lower bound of the interval. Null hypotheses have limited usefulness even at p=0.005. A tiny systematic bias in the experiment can make it cross that threshold and there are _always_ at least small biases. These can be caused by not exactly calibrated instruments or small differences in the way tests are performed by different researchers on the team etc. Inherent in null hypothesis testing is this ridiculous assumption that there are no systematic biases. This is its fatal flaw. The null hypothesis test never tells you that the effect is zero or that it is any other value, it only gives you an hint about whether or not the data is too noisy to say anything at all.

Retric 9y ago
"There is a 95% chance that this data isn't pure noise". That's not what P=0.05 actually means. It's much closer to, this result rises above the noise floor. Suggesting but not implying a possible link between X and Y, fell free to investigate.
tedsanders 9y ago
>P=0.05 is best interpreted as "There is a 95% chance that this data isn't pure noise". No, that's not what P=0.05 means. This misconception is exactly why p values are so problematic. The question we want to answer is unanswerable, so instead we subtly substitute a different question in its place and don't even realize we've swapped concepts.
- BenoitEssiambre 9y ago
  When I say that "There is a 95% chance that this data isn't pure noise" I'm being slightly indirect. More directly it would be something like "There is 5% chance that pure noise generator you would give you this result" but since I want to reason about the actual world that the experiment is trying to study, I think it is legitimate to do the Bayesian flip and infer the more useful first statement that the world is likely too noisy to detect an effect using this experiment.
  - andy_wrote 9y ago
    I think I understand what you're trying to get at, but I don't think describing a p-value in terms of "pure noise" is quite right. The null hypothesis may well carry with it all sorts of specific assumptions: independence, finite variance, maybe even specific probability distributions (and not necessarily the normal distribution). Typically somewhere in there we define a θ parameter and then include something like θ = 0 that provides grounds for possible rejection. We think of "θ = 0" as the null hypothesis, but all the other assumptions are part of the null hypothesis as well; the statement "θ = 0" does not make sense without them.
  - nonbel 9y ago
    If you see clouds how likely is rain? If you see rain how likely is it there are clouds? Are the answers to these at all the same? No. There is absolutely nothing valid about your "flip" (called transposing the conditional) and the damage done to the human species by people doing the "flip" is too huge for most to comprehend. Sorry, but I really cannot overstate the problems with this "flip". Perhaps people who get paid by grants should get three chances regarding this flip. If they strike out they are banned from ever being funded by NIH, etc for life. The result (either 99+% of current researchers would be culled, or p-values would become appropriately rare) would do more to advance science than any other idea I have seen.
    - BenoitEssiambre 9y ago
      If you want your science to be about the real world and not about some hypothetical scenarios that don't exist you have no choice but to do the transposition. Experiments are useless otherwise. There is a reason humans naturally and instinctively do it when they talk about experiments. This is how you describe things in real world human language. Sure there are debates about priors. These are unavoidable. The hidden assumption in my example is a 50/50 prior on too noisy, not too noisy hypotheses. IMO a reasonable uniform (maximum entropy) prior to start a discussion from. Of course, one should be more precise when doing this type of analysis in a paper. The reason a good prior is hard to define here is not because you shouldn't do the flip, it is because the question that the t-test is trying to answer doesn't make much sense.
ak_yo 9y ago
> P=0.05 is best interpreted as "There is a 95% chance that this data isn't pure noise". This is incorrect. A p-value is the probability of observing a value of the test statistic that is at least as extreme as the reported value, assuming the null hypothesis is true.
- nils-m-holm 9y ago
  Which sounds like a fancy way of saying that "there is a 5% chance of the data being noise (while the null hypothesis is true)".
  - imh 9y ago
    "there is a 5% chance of the data being noise" is a statement on the probability of whether the phenomenon is not real given the data: p(it's just noise | data). A p value is a statement on the probability we'd see this data given that the phenomenon isn't real: p(data | it's just noise). Now, the exact values of a given data set are vanishingly unlikely, so we actually ask about the probability we'd say this data or even more extreme data given that we're just looking at noise. That's a typical p value.
    - nils-m-holm 9y ago
      That's what I meant: P(data | H0). So there is a 5% probability for the data to arise by chance (being noise) given H0 is true. The whole terminology with values being "extreme" is just unintuitive and unwieldy, IMO.

Retric 9y ago

"There is a 95% chance that this data isn't pure noise". That's not what P=0.05 actually means. It's much closer to, this result rises above the noise floor. Suggesting but not implying a possible link between X and Y, fell free to investigate.

tedsanders 9y ago

>P=0.05 is best interpreted as "There is a 95% chance that this data isn't pure noise". No, that's not what P=0.05 means. This misconception is exactly why p values are so problematic. The question we want to answer is unanswerable, so instead we subtly substitute a different question in its place and don't even realize we've swapped concepts.

BenoitEssiambre 9y ago
When I say that "There is a 95% chance that this data isn't pure noise" I'm being slightly indirect. More directly it would be something like "There is 5% chance that pure noise generator you would give you this result" but since I want to reason about the actual world that the experiment is trying to study, I think it is legitimate to do the Bayesian flip and infer the more useful first statement that the world is likely too noisy to detect an effect using this experiment.
- andy_wrote 9y ago
  I think I understand what you're trying to get at, but I don't think describing a p-value in terms of "pure noise" is quite right. The null hypothesis may well carry with it all sorts of specific assumptions: independence, finite variance, maybe even specific probability distributions (and not necessarily the normal distribution). Typically somewhere in there we define a θ parameter and then include something like θ = 0 that provides grounds for possible rejection. We think of "θ = 0" as the null hypothesis, but all the other assumptions are part of the null hypothesis as well; the statement "θ = 0" does not make sense without them.
- nonbel 9y ago
  If you see clouds how likely is rain? If you see rain how likely is it there are clouds? Are the answers to these at all the same? No. There is absolutely nothing valid about your "flip" (called transposing the conditional) and the damage done to the human species by people doing the "flip" is too huge for most to comprehend. Sorry, but I really cannot overstate the problems with this "flip". Perhaps people who get paid by grants should get three chances regarding this flip. If they strike out they are banned from ever being funded by NIH, etc for life. The result (either 99+% of current researchers would be culled, or p-values would become appropriately rare) would do more to advance science than any other idea I have seen.
  - BenoitEssiambre 9y ago
    If you want your science to be about the real world and not about some hypothetical scenarios that don't exist you have no choice but to do the transposition. Experiments are useless otherwise. There is a reason humans naturally and instinctively do it when they talk about experiments. This is how you describe things in real world human language. Sure there are debates about priors. These are unavoidable. The hidden assumption in my example is a 50/50 prior on too noisy, not too noisy hypotheses. IMO a reasonable uniform (maximum entropy) prior to start a discussion from. Of course, one should be more precise when doing this type of analysis in a paper. The reason a good prior is hard to define here is not because you shouldn't do the flip, it is because the question that the t-test is trying to answer doesn't make much sense.

ak_yo 9y ago

> P=0.05 is best interpreted as "There is a 95% chance that this data isn't pure noise". This is incorrect. A p-value is the probability of observing a value of the test statistic that is at least as extreme as the reported value, assuming the null hypothesis is true.

nils-m-holm 9y ago
Which sounds like a fancy way of saying that "there is a 5% chance of the data being noise (while the null hypothesis is true)".
- imh 9y ago
  "there is a 5% chance of the data being noise" is a statement on the probability of whether the phenomenon is not real given the data: p(it's just noise | data). A p value is a statement on the probability we'd see this data given that the phenomenon isn't real: p(data | it's just noise). Now, the exact values of a given data set are vanishingly unlikely, so we actually ask about the probability we'd say this data or even more extreme data given that we're just looking at noise. That's a typical p value.
  - nils-m-holm 9y ago
    That's what I meant: P(data | H0). So there is a 5% probability for the data to arise by chance (being noise) given H0 is true. The whole terminology with values being "extreme" is just unintuitive and unwieldy, IMO.