3 ms·

If the null-hypothesis is true, every observation made should be consistent with it. This will result in P(X|H) trending to 1 (since there will be experimental

If the null-hypothesis is true, every observation made should be consistent with it. This will result in P(X|H) trending to 1 (since there will be experimental variance). No other experimental design makes sense. >The meaning [of p-value when H is false] is clear: "the probability of getting a value for the statistic as high as the observed one if the null hypothesis was true" This is a logical fallacy. It is counterfactual to consider a world where the null hypothesis is true, when it is not. In fact, this is precisely the feature of the universe that p-value based experimentation exploits and is essentially the only way for us to gain any information about 'reality'. >By definition, if the null hypothesis is true the p-value is uniformly distributed between 0 and 1. I don't think so. If that were true, p-value would be entirely useless.

dragonwriter 11y ago
> If the null-hypothesis is true, every observation made should be consistent with it. This will result in P(X|H) trending to 1 Every observation being consistent with H doesn't mean that for each event X that occurs, the conditional probability of X given H will be, or "trend toward", 1. Assuming a perfectly deterministic universe, the P(X|everything else that is true) will be 1 for every X that occurs, but that doesn't mean P(X|H) for any particular true proposition H will be anything like that.
- jsprogrammer 11y ago
  >Every observation being consistent with H doesn't mean that for each event X that occurs, the conditional probability of X given H will be, or "trend toward", 1. Agreed. If you plot P(X|H) (computed over all observations) over time, in a well-formed experiment the line will trend to 1 if the null-hypothesis is true and to 0 if the null-hypothesis is false. It really is that simple.
kgwgk 11y ago
You definitely do not know what a p-value is. When you wrote "P(X|H)" I though X was shorthand for T>T(X) where T is the statistic, not that you were referring to the actual data X. P(X|H) doesn't have the properties you claim, anyway. P(X|H)=1 corresponds to the case where only one outcome is possible. In non-trivial cases, the more data you add the lower this number will be. Assume H="you have a fair coin". You throw it once: heads. P(X|H)=P(h|faircoin)=1/2 You throw it again: tails. P(X|H)=P(ht|faircoin)=1/4. You throw it again: tails. P(X|H)=P(htt|faircoin)=1/8. I guess the experiment is not well-formed...
- jsprogrammer 11y ago
  X can be any random variable that satisfies the requirements of the null-hypothesis. A more appropriate variable for your experiment would probably be the ratio of heads to tails (may need to add a bias to avoid division by 0). "you have a fair coin" is not a hypothesis, at least not a well-defined one.
  - kgwgk 11y ago
    Ok, so you're thinking about a random variable which converges to some value when the null hypothesis is true. This is fine, but it has nothing to do whatsoever with p-values. Let me say that your notation is not very appropriate. It makes no sense to say that P(X|H) converges to 1. If you expect X to converge to C if the null hypothesis is true, you can simply say X->C. A proper notation involving probabilities would be P(|X-C|>epsilon)->0 for any positive epsilon (convergence in probability) or maybe P(X->C)=1 (convergence almost surely). Taking as you suggest X=(#tails/#heads), you expect that X->1 if the coin is fair (I'm not sure why you find this is not a well-defined null hypothesis, but I don't really care). However, P(X)<1 for every X. In fact, P(X=1)->0 as the number if trials increases (X will get closer to 1 on average, but getting exactly 1 will get more and more unlikely). As I said, you're free to prefer your converging statistics and your well-defined null hypothesis. But you should be aware that people are talking about something completely different when discussing things like the 1e-7 p-value in the Higgs boson discovery or the reproducibility of statistically significant results. EDIT: Another example, maybe better-defined: a random variable distributed (under the null hypothesis) x~Normal(mu=0,sigma=1). Let's say you take N samples (I let you choose the number, so I don't pick one which is not good enough).The statistic is the mean X=(x_1+x_2+..+x_N)/N. If the null hypothesis is true, X->mu=0. You get X=1/sqrt(N). What's your "p-value" in that case?
    - jsprogrammer 11y ago
      >Ok, so you're thinking about a random variable which converges to some value when the null hypothesis is true. This is fine, but it has nothing to do whatsoever with p-values. Well, the variable itself doesn't, just the observed value of P(X|H). X can be any random variable, but typically it will need to be transformed to have a normal distribution about 0 with a standard deviation of 1 (since this is what the typical null-hypothesis predicts). To effectively use p-value analysis, it is typically assumed that your null-hypothesis predicts that your observations will be normally distributed with a mean of 0 and a standard deviation of 1. The total count of heads observed will not be distributed that way. Neither will the probability of a particular sequence (what your example seemed to be calculating). I say your null hypothesis is not well-defined because the term 'fair' remains undefined (though we could guess at the meaning) and in fact makes no predictions about the world. You need to apply transformations to your random variable so that it will appear normally distributed about 0 with a standard deviation of 1 if the hypothesis is true. >Let me say that your notation is not very appropriate. It makes no sense to say that P(X|H) converges to 1. My notation is perfectly appropriate. X is a random variable and a random variable is the only thing that can go there (if you are doing p-value analysis). X is not assumed to be uniform or simple (although it certainly could be). P(T>T(X)|H) can be replaced with P(Y|H) every time (Y = T>T(X)). >As I said, you're free to prefer your converging statistics and your well-defined null hypothesis. But you should be aware that people are talking about something completely different when discussing things like the 1e-7 p-value in the Higgs boson discovery or the reproducibility of statistically significant results. I'm glad that we finally agree on this (although I dispute that anyone working on the Higgs boson discovery disagrees with me). One of my first claims was that others may not be calculating true p-values, but may calculate something and call it 'p-value' and then think that it means something it does not. In fact, this entire topic even links to an article in a prominent publisher claiming the same. Do you think it is purely coincidental that the figures I showed you from the Higgs experiment show the lines converging towards only two different numbers: 1 and 0? Edit: You'll have to give me some time on your edit. It's not something I typically calculate and I have other business to attend to today.

dragonwriter 11y ago

> If the null-hypothesis is true, every observation made should be consistent with it. This will result in P(X|H) trending to 1 Every observation being consistent with H doesn't mean that for each event X that occurs, the conditional probability of X given H will be, or "trend toward", 1. Assuming a perfectly deterministic universe, the P(X|everything else that is true) will be 1 for every X that occurs, but that doesn't mean P(X|H) for any particular true proposition H will be anything like that.

jsprogrammer 11y ago
>Every observation being consistent with H doesn't mean that for each event X that occurs, the conditional probability of X given H will be, or "trend toward", 1. Agreed. If you plot P(X|H) (computed over all observations) over time, in a well-formed experiment the line will trend to 1 if the null-hypothesis is true and to 0 if the null-hypothesis is false. It really is that simple.

kgwgk 11y ago

You definitely do not know what a p-value is. When you wrote "P(X|H)" I though X was shorthand for T>T(X) where T is the statistic, not that you were referring to the actual data X. P(X|H) doesn't have the properties you claim, anyway. P(X|H)=1 corresponds to the case where only one outcome is possible. In non-trivial cases, the more data you add the lower this number will be. Assume H="you have a fair coin". You throw it once: heads. P(X|H)=P(h|faircoin)=1/2 You throw it again: tails. P(X|H)=P(ht|faircoin)=1/4. You throw it again: tails. P(X|H)=P(htt|faircoin)=1/8. I guess the experiment is not well-formed...

jsprogrammer 11y ago
X can be any random variable that satisfies the requirements of the null-hypothesis. A more appropriate variable for your experiment would probably be the ratio of heads to tails (may need to add a bias to avoid division by 0). "you have a fair coin" is not a hypothesis, at least not a well-defined one.
- kgwgk 11y ago
  Ok, so you're thinking about a random variable which converges to some value when the null hypothesis is true. This is fine, but it has nothing to do whatsoever with p-values. Let me say that your notation is not very appropriate. It makes no sense to say that P(X|H) converges to 1. If you expect X to converge to C if the null hypothesis is true, you can simply say X->C. A proper notation involving probabilities would be P(|X-C|>epsilon)->0 for any positive epsilon (convergence in probability) or maybe P(X->C)=1 (convergence almost surely). Taking as you suggest X=(#tails/#heads), you expect that X->1 if the coin is fair (I'm not sure why you find this is not a well-defined null hypothesis, but I don't really care). However, P(X)<1 for every X. In fact, P(X=1)->0 as the number if trials increases (X will get closer to 1 on average, but getting exactly 1 will get more and more unlikely). As I said, you're free to prefer your converging statistics and your well-defined null hypothesis. But you should be aware that people are talking about something completely different when discussing things like the 1e-7 p-value in the Higgs boson discovery or the reproducibility of statistically significant results. EDIT: Another example, maybe better-defined: a random variable distributed (under the null hypothesis) x~Normal(mu=0,sigma=1). Let's say you take N samples (I let you choose the number, so I don't pick one which is not good enough).The statistic is the mean X=(x_1+x_2+..+x_N)/N. If the null hypothesis is true, X->mu=0. You get X=1/sqrt(N). What's your "p-value" in that case?
  - jsprogrammer 11y ago
    >Ok, so you're thinking about a random variable which converges to some value when the null hypothesis is true. This is fine, but it has nothing to do whatsoever with p-values. Well, the variable itself doesn't, just the observed value of P(X|H). X can be any random variable, but typically it will need to be transformed to have a normal distribution about 0 with a standard deviation of 1 (since this is what the typical null-hypothesis predicts). To effectively use p-value analysis, it is typically assumed that your null-hypothesis predicts that your observations will be normally distributed with a mean of 0 and a standard deviation of 1. The total count of heads observed will not be distributed that way. Neither will the probability of a particular sequence (what your example seemed to be calculating). I say your null hypothesis is not well-defined because the term 'fair' remains undefined (though we could guess at the meaning) and in fact makes no predictions about the world. You need to apply transformations to your random variable so that it will appear normally distributed about 0 with a standard deviation of 1 if the hypothesis is true. >Let me say that your notation is not very appropriate. It makes no sense to say that P(X|H) converges to 1. My notation is perfectly appropriate. X is a random variable and a random variable is the only thing that can go there (if you are doing p-value analysis). X is not assumed to be uniform or simple (although it certainly could be). P(T>T(X)|H) can be replaced with P(Y|H) every time (Y = T>T(X)). >As I said, you're free to prefer your converging statistics and your well-defined null hypothesis. But you should be aware that people are talking about something completely different when discussing things like the 1e-7 p-value in the Higgs boson discovery or the reproducibility of statistically significant results. I'm glad that we finally agree on this (although I dispute that anyone working on the Higgs boson discovery disagrees with me). One of my first claims was that others may not be calculating true p-values, but may calculate something and call it 'p-value' and then think that it means something it does not. In fact, this entire topic even links to an article in a prominent publisher claiming the same. Do you think it is purely coincidental that the figures I showed you from the Higgs experiment show the lines converging towards only two different numbers: 1 and 0? Edit: You'll have to give me some time on your edit. It's not something I typically calculate and I have other business to attend to today.