3 ms·

Well, before you go, I implore you to look into the actual computation and theory of 'p-value'. A p-value is simply P(X|H). P(X|H) only means something when H

Well, before you go, I implore you to look into the actual computation and theory of 'p-value'. A p-value is simply P(X|H). P(X|H) only means something when H is true. If H is false, P(X|H) tells you nothing. Since H is your null-hypothesis, if it does not actually hold in the real-world, P(X|H) is meaningless. If you read the paper I linked, they never explicitly call out the null hypothesis (nor do, I believe, they show the work for their calculations). There should be another paper somewhere that describes exactly what it is, in the terms I am using. So, phrases like, "[t]hey don't see any problem in stating a p-value and rejecting the null hypothesis" make me think you have no idea what you're talking about. The null hypothesis can never be 'rejected' (ie. p-value can never reach 0). I don't think you will find anyone working on the Higgs boson that will claim otherwise.

kgwgk 11y ago
I think we agree that their null hypothesis is "there is a background, with events coming from all the known particles". I think we agree that their conclusion is "these results provide conclusive evidence for the discovery of a new particle". I don't see how can they say that there is a new particle without rejecting the hypothesis that there is no such new particle. Of course you can say that the null hypothesis can never be rejected (relevant Dilbert strip: http://dilbert.com/strip/2001-10-25 http://dilbert.com/strip/2001-10-25) but then they can never discover a new particle either. Regarding p-values in general, your definition is the same I've been using all along. But I don't think it is meaningless when the null hypothesis does not hold. The meaning is clear: "the probability of getting a value for the statistic as high as the observed one if the null hypothesis was true". For example, there would be one chance in several millions of observing the kind of data they found at the LHC if the Higgs boson didn't exist. You might want to look into the theory yourself, because the notion of p-values trending towards 1 if the null hypothesis is true is nonsense. By definition, if the null hypothesis is true the p-value is uniformly distributed between 0 and 1. If you have at some point a p-value close to one (or to any other number for that matter) and keep adding data, in the long run it will still be uniformly distributed between 0 and 1.
- jsprogrammer 11y ago
  If the null-hypothesis is true, every observation made should be consistent with it. This will result in P(X|H) trending to 1 (since there will be experimental variance). No other experimental design makes sense. >The meaning [of p-value when H is false] is clear: "the probability of getting a value for the statistic as high as the observed one if the null hypothesis was true" This is a logical fallacy. It is counterfactual to consider a world where the null hypothesis is true, when it is not. In fact, this is precisely the feature of the universe that p-value based experimentation exploits and is essentially the only way for us to gain any information about 'reality'. >By definition, if the null hypothesis is true the p-value is uniformly distributed between 0 and 1. I don't think so. If that were true, p-value would be entirely useless.
  - dragonwriter 11y ago
    > If the null-hypothesis is true, every observation made should be consistent with it. This will result in P(X|H) trending to 1 Every observation being consistent with H doesn't mean that for each event X that occurs, the conditional probability of X given H will be, or "trend toward", 1. Assuming a perfectly deterministic universe, the P(X|everything else that is true) will be 1 for every X that occurs, but that doesn't mean P(X|H) for any particular true proposition H will be anything like that.
    - jsprogrammer 11y ago
      >Every observation being consistent with H doesn't mean that for each event X that occurs, the conditional probability of X given H will be, or "trend toward", 1. Agreed. If you plot P(X|H) (computed over all observations) over time, in a well-formed experiment the line will trend to 1 if the null-hypothesis is true and to 0 if the null-hypothesis is false. It really is that simple.
  - kgwgk 11y ago
    You definitely do not know what a p-value is. When you wrote "P(X|H)" I though X was shorthand for T>T(X) where T is the statistic, not that you were referring to the actual data X. P(X|H) doesn't have the properties you claim, anyway. P(X|H)=1 corresponds to the case where only one outcome is possible. In non-trivial cases, the more data you add the lower this number will be. Assume H="you have a fair coin". You throw it once: heads. P(X|H)=P(h|faircoin)=1/2 You throw it again: tails. P(X|H)=P(ht|faircoin)=1/4. You throw it again: tails. P(X|H)=P(htt|faircoin)=1/8. I guess the experiment is not well-formed...
dragonwriter 11y ago
> A p-value is simply P(X|H). P(X|H) only means something when H is true. If H is false, P(X|H) tells you nothing. If you know H is false, P(X|H) tells you nothing. But then H wouldn't be a hypothesis, null or otherwise. If you don't know whether H is true, but you do know something about X, P(X|H) tells you something useful about whether the positive hypothesis to which H is the alternative has an effect apparent in the world to explain. > The null hypothesis can never be 'rejected' (ie. p-value can never reach 0). Rejection of the null hypothesis does not mean p-value = 0. Scientific progress is not based on logical certainty, but rather practical utility. Necessary truths are the domain of pure logic, not empirical science.
- jsprogrammer 11y ago
  What is the interpretation of a p-value = 0 then? Empirical science can never reject any theory, it is not powerful enough. At best it can provide a selection of least worst explanations. The H in P(X|H) does not mean 'assumed to be true', it means 'is in fact true'. If H is in fact false, it is counter-factual to assume it is true and therefore any conclusions drawn from the assumption are invalid. This is independent of belief in H. >Necessary truths are the domain of pure logic, not empirical science. Science can never deliver truth, which is why it can never truly reject anything (including null-hypotheses). More generally, this is referred to as the problem of induction.

kgwgk 11y ago

I think we agree that their null hypothesis is "there is a background, with events coming from all the known particles". I think we agree that their conclusion is "these results provide conclusive evidence for the discovery of a new particle". I don't see how can they say that there is a new particle without rejecting the hypothesis that there is no such new particle. Of course you can say that the null hypothesis can never be rejected (relevant Dilbert strip: http://dilbert.com/strip/2001-10-25 http://dilbert.com/strip/2001-10-25) but then they can never discover a new particle either. Regarding p-values in general, your definition is the same I've been using all along. But I don't think it is meaningless when the null hypothesis does not hold. The meaning is clear: "the probability of getting a value for the statistic as high as the observed one if the null hypothesis was true". For example, there would be one chance in several millions of observing the kind of data they found at the LHC if the Higgs boson didn't exist. You might want to look into the theory yourself, because the notion of p-values trending towards 1 if the null hypothesis is true is nonsense. By definition, if the null hypothesis is true the p-value is uniformly distributed between 0 and 1. If you have at some point a p-value close to one (or to any other number for that matter) and keep adding data, in the long run it will still be uniformly distributed between 0 and 1.

jsprogrammer 11y ago
If the null-hypothesis is true, every observation made should be consistent with it. This will result in P(X|H) trending to 1 (since there will be experimental variance). No other experimental design makes sense. >The meaning [of p-value when H is false] is clear: "the probability of getting a value for the statistic as high as the observed one if the null hypothesis was true" This is a logical fallacy. It is counterfactual to consider a world where the null hypothesis is true, when it is not. In fact, this is precisely the feature of the universe that p-value based experimentation exploits and is essentially the only way for us to gain any information about 'reality'. >By definition, if the null hypothesis is true the p-value is uniformly distributed between 0 and 1. I don't think so. If that were true, p-value would be entirely useless.
- dragonwriter 11y ago
  > If the null-hypothesis is true, every observation made should be consistent with it. This will result in P(X|H) trending to 1 Every observation being consistent with H doesn't mean that for each event X that occurs, the conditional probability of X given H will be, or "trend toward", 1. Assuming a perfectly deterministic universe, the P(X|everything else that is true) will be 1 for every X that occurs, but that doesn't mean P(X|H) for any particular true proposition H will be anything like that.
  - jsprogrammer 11y ago
    >Every observation being consistent with H doesn't mean that for each event X that occurs, the conditional probability of X given H will be, or "trend toward", 1. Agreed. If you plot P(X|H) (computed over all observations) over time, in a well-formed experiment the line will trend to 1 if the null-hypothesis is true and to 0 if the null-hypothesis is false. It really is that simple.
- kgwgk 11y ago
  You definitely do not know what a p-value is. When you wrote "P(X|H)" I though X was shorthand for T>T(X) where T is the statistic, not that you were referring to the actual data X. P(X|H) doesn't have the properties you claim, anyway. P(X|H)=1 corresponds to the case where only one outcome is possible. In non-trivial cases, the more data you add the lower this number will be. Assume H="you have a fair coin". You throw it once: heads. P(X|H)=P(h|faircoin)=1/2 You throw it again: tails. P(X|H)=P(ht|faircoin)=1/4. You throw it again: tails. P(X|H)=P(htt|faircoin)=1/8. I guess the experiment is not well-formed...

dragonwriter 11y ago

> A p-value is simply P(X|H). P(X|H) only means something when H is true. If H is false, P(X|H) tells you nothing. If you know H is false, P(X|H) tells you nothing. But then H wouldn't be a hypothesis, null or otherwise. If you don't know whether H is true, but you do know something about X, P(X|H) tells you something useful about whether the positive hypothesis to which H is the alternative has an effect apparent in the world to explain. > The null hypothesis can never be 'rejected' (ie. p-value can never reach 0). Rejection of the null hypothesis does not mean p-value = 0. Scientific progress is not based on logical certainty, but rather practical utility. Necessary truths are the domain of pure logic, not empirical science.

jsprogrammer 11y ago
What is the interpretation of a p-value = 0 then? Empirical science can never reject any theory, it is not powerful enough. At best it can provide a selection of least worst explanations. The H in P(X|H) does not mean 'assumed to be true', it means 'is in fact true'. If H is in fact false, it is counter-factual to assume it is true and therefore any conclusions drawn from the assumption are invalid. This is independent of belief in H. >Necessary truths are the domain of pure logic, not empirical science. Science can never deliver truth, which is why it can never truly reject anything (including null-hypotheses). More generally, this is referred to as the problem of induction.