17 ms·

When I say that "There is a 95% chance that this data isn't pure noise" I'm being slightly indirect. More directly it would be something like "There is 5% chanc

When I say that "There is a 95% chance that this data isn't pure noise" I'm being slightly indirect. More directly it would be something like "There is 5% chance that pure noise generator you would give you this result" but since I want to reason about the actual world that the experiment is trying to study, I think it is legitimate to do the Bayesian flip and infer the more useful first statement that the world is likely too noisy to detect an effect using this experiment.

andy_wrote 9y ago
I think I understand what you're trying to get at, but I don't think describing a p-value in terms of "pure noise" is quite right. The null hypothesis may well carry with it all sorts of specific assumptions: independence, finite variance, maybe even specific probability distributions (and not necessarily the normal distribution). Typically somewhere in there we define a θ parameter and then include something like θ = 0 that provides grounds for possible rejection. We think of "θ = 0" as the null hypothesis, but all the other assumptions are part of the null hypothesis as well; the statement "θ = 0" does not make sense without them.
nonbel 9y ago
If you see clouds how likely is rain? If you see rain how likely is it there are clouds? Are the answers to these at all the same? No. There is absolutely nothing valid about your "flip" (called transposing the conditional) and the damage done to the human species by people doing the "flip" is too huge for most to comprehend. Sorry, but I really cannot overstate the problems with this "flip". Perhaps people who get paid by grants should get three chances regarding this flip. If they strike out they are banned from ever being funded by NIH, etc for life. The result (either 99+% of current researchers would be culled, or p-values would become appropriately rare) would do more to advance science than any other idea I have seen.
- BenoitEssiambre 9y ago
  If you want your science to be about the real world and not about some hypothetical scenarios that don't exist you have no choice but to do the transposition. Experiments are useless otherwise. There is a reason humans naturally and instinctively do it when they talk about experiments. This is how you describe things in real world human language. Sure there are debates about priors. These are unavoidable. The hidden assumption in my example is a 50/50 prior on too noisy, not too noisy hypotheses. IMO a reasonable uniform (maximum entropy) prior to start a discussion from. Of course, one should be more precise when doing this type of analysis in a paper. The reason a good prior is hard to define here is not because you shouldn't do the flip, it is because the question that the t-test is trying to answer doesn't make much sense.
  - nonbel 9y ago
    >"If you want your science to be about the real world and not about some hypothetical scenarios that don't exist you have no choice but to do the transposition." How can saying false things make your science "about the the real world"? The only thing I can think of is that by "real world" you are referring to using the publication of massive amounts of incorrect/questionable information as a metric for success. Yes, that is indeed the sham going on. >"There is a reason humans naturally and instinctively do it when they talk about experiments. This is how you describe things in real world human language." I have no problem talking about data without transposing the conditional, if you do that is some issue with your training. >"Sure there are debates about priors." The error you advocate (of transposing the conditional) has nothing to do with priors.
    - BenoitEssiambre 9y ago
      There is no point in talking about the data if it doesn't tell you useful things about the population or type of thing you are studying, if it's not for building a predictive model of the subjects. This is the entire reason for research. Also P(A|B) = P(B|A) if the prior probabilities are equal, often a reasonable prior.
      - nonbel 9y ago
        >"There is no point in talking about the data if it doesn't tell you useful things about the population or type of thing you are studying" Ok, but I fail to see the relevance of this to anything. >"Also P(A|B) = P(B|A) if the prior probabilities are equal, often a reasonable prior." Here, A = Hypothesis and B = data (or vice versa). You are claiming that the "probability of seeing your data whether or not the hypothesis is correct", is often near the "probability the hypothesis is correct independent of your data"? Based on what? What principle leads you to think these two probabilities should be near each other? BTW, if you actually have one this is like the holy grail of statistics.
        Bromskloss 9y ago

andy_wrote 9y ago

I think I understand what you're trying to get at, but I don't think describing a p-value in terms of "pure noise" is quite right. The null hypothesis may well carry with it all sorts of specific assumptions: independence, finite variance, maybe even specific probability distributions (and not necessarily the normal distribution). Typically somewhere in there we define a θ parameter and then include something like θ = 0 that provides grounds for possible rejection. We think of "θ = 0" as the null hypothesis, but all the other assumptions are part of the null hypothesis as well; the statement "θ = 0" does not make sense without them.

nonbel 9y ago

If you see clouds how likely is rain? If you see rain how likely is it there are clouds? Are the answers to these at all the same? No. There is absolutely nothing valid about your "flip" (called transposing the conditional) and the damage done to the human species by people doing the "flip" is too huge for most to comprehend. Sorry, but I really cannot overstate the problems with this "flip". Perhaps people who get paid by grants should get three chances regarding this flip. If they strike out they are banned from ever being funded by NIH, etc for life. The result (either 99+% of current researchers would be culled, or p-values would become appropriately rare) would do more to advance science than any other idea I have seen.

BenoitEssiambre 9y ago
If you want your science to be about the real world and not about some hypothetical scenarios that don't exist you have no choice but to do the transposition. Experiments are useless otherwise. There is a reason humans naturally and instinctively do it when they talk about experiments. This is how you describe things in real world human language. Sure there are debates about priors. These are unavoidable. The hidden assumption in my example is a 50/50 prior on too noisy, not too noisy hypotheses. IMO a reasonable uniform (maximum entropy) prior to start a discussion from. Of course, one should be more precise when doing this type of analysis in a paper. The reason a good prior is hard to define here is not because you shouldn't do the flip, it is because the question that the t-test is trying to answer doesn't make much sense.
- nonbel 9y ago
  >"If you want your science to be about the real world and not about some hypothetical scenarios that don't exist you have no choice but to do the transposition." How can saying false things make your science "about the the real world"? The only thing I can think of is that by "real world" you are referring to using the publication of massive amounts of incorrect/questionable information as a metric for success. Yes, that is indeed the sham going on. >"There is a reason humans naturally and instinctively do it when they talk about experiments. This is how you describe things in real world human language." I have no problem talking about data without transposing the conditional, if you do that is some issue with your training. >"Sure there are debates about priors." The error you advocate (of transposing the conditional) has nothing to do with priors.
  - BenoitEssiambre 9y ago
    There is no point in talking about the data if it doesn't tell you useful things about the population or type of thing you are studying, if it's not for building a predictive model of the subjects. This is the entire reason for research. Also P(A|B) = P(B|A) if the prior probabilities are equal, often a reasonable prior.
    - nonbel 9y ago
      >"There is no point in talking about the data if it doesn't tell you useful things about the population or type of thing you are studying" Ok, but I fail to see the relevance of this to anything. >"Also P(A|B) = P(B|A) if the prior probabilities are equal, often a reasonable prior." Here, A = Hypothesis and B = data (or vice versa). You are claiming that the "probability of seeing your data whether or not the hypothesis is correct", is often near the "probability the hypothesis is correct independent of your data"? Based on what? What principle leads you to think these two probabilities should be near each other? BTW, if you actually have one this is like the holy grail of statistics.
      - Bromskloss 9y ago