4 ms·

X can be any random variable that satisfies the requirements of the null-hypothesis. A more appropriate variable for your experiment would probably be the rati

X can be any random variable that satisfies the requirements of the null-hypothesis. A more appropriate variable for your experiment would probably be the ratio of heads to tails (may need to add a bias to avoid division by 0). "you have a fair coin" is not a hypothesis, at least not a well-defined one.

kgwgk 11y ago
Ok, so you're thinking about a random variable which converges to some value when the null hypothesis is true. This is fine, but it has nothing to do whatsoever with p-values. Let me say that your notation is not very appropriate. It makes no sense to say that P(X|H) converges to 1. If you expect X to converge to C if the null hypothesis is true, you can simply say X->C. A proper notation involving probabilities would be P(|X-C|>epsilon)->0 for any positive epsilon (convergence in probability) or maybe P(X->C)=1 (convergence almost surely). Taking as you suggest X=(#tails/#heads), you expect that X->1 if the coin is fair (I'm not sure why you find this is not a well-defined null hypothesis, but I don't really care). However, P(X)<1 for every X. In fact, P(X=1)->0 as the number if trials increases (X will get closer to 1 on average, but getting exactly 1 will get more and more unlikely). As I said, you're free to prefer your converging statistics and your well-defined null hypothesis. But you should be aware that people are talking about something completely different when discussing things like the 1e-7 p-value in the Higgs boson discovery or the reproducibility of statistically significant results. EDIT: Another example, maybe better-defined: a random variable distributed (under the null hypothesis) x~Normal(mu=0,sigma=1). Let's say you take N samples (I let you choose the number, so I don't pick one which is not good enough).The statistic is the mean X=(x_1+x_2+..+x_N)/N. If the null hypothesis is true, X->mu=0. You get X=1/sqrt(N). What's your "p-value" in that case?
- jsprogrammer 11y ago
  >Ok, so you're thinking about a random variable which converges to some value when the null hypothesis is true. This is fine, but it has nothing to do whatsoever with p-values. Well, the variable itself doesn't, just the observed value of P(X|H). X can be any random variable, but typically it will need to be transformed to have a normal distribution about 0 with a standard deviation of 1 (since this is what the typical null-hypothesis predicts). To effectively use p-value analysis, it is typically assumed that your null-hypothesis predicts that your observations will be normally distributed with a mean of 0 and a standard deviation of 1. The total count of heads observed will not be distributed that way. Neither will the probability of a particular sequence (what your example seemed to be calculating). I say your null hypothesis is not well-defined because the term 'fair' remains undefined (though we could guess at the meaning) and in fact makes no predictions about the world. You need to apply transformations to your random variable so that it will appear normally distributed about 0 with a standard deviation of 1 if the hypothesis is true. >Let me say that your notation is not very appropriate. It makes no sense to say that P(X|H) converges to 1. My notation is perfectly appropriate. X is a random variable and a random variable is the only thing that can go there (if you are doing p-value analysis). X is not assumed to be uniform or simple (although it certainly could be). P(T>T(X)|H) can be replaced with P(Y|H) every time (Y = T>T(X)). >As I said, you're free to prefer your converging statistics and your well-defined null hypothesis. But you should be aware that people are talking about something completely different when discussing things like the 1e-7 p-value in the Higgs boson discovery or the reproducibility of statistically significant results. I'm glad that we finally agree on this (although I dispute that anyone working on the Higgs boson discovery disagrees with me). One of my first claims was that others may not be calculating true p-values, but may calculate something and call it 'p-value' and then think that it means something it does not. In fact, this entire topic even links to an article in a prominent publisher claiming the same. Do you think it is purely coincidental that the figures I showed you from the Higgs experiment show the lines converging towards only two different numbers: 1 and 0? Edit: You'll have to give me some time on your edit. It's not something I typically calculate and I have other business to attend to today.
  - kgwgk 11y ago
    Ok, so maybe your definition does correspond to a p-value after all. It's hard to say as you have refused to discuss concrete cases (like the fair coin or the loaded die, which are standard examples to introduce p-values). But if you're actually calculating a p-value then it won't behave as you expect. It won't converge to anything (edit: if the null hypothesis holds). P-values are by definition uniformly distributed when the null hypothesis is true. If your "p-value" is not, then it's not a p-value. It really is that simple. Or maybe everyone else is using the wrong "p-values" and yours are the real thing. You can believe it if you want. Please disregard my previous questions, I see no point in continuing this discussion. But you might want to read a bit more about p-values: you won't find anyone (I hope!) sharing your point of view. Once you understand what the p-value is, and what it is not, you might indeed conclude that they are entirely useless. Of course it's your right to avoid learning what p-values really are, and keep the faith. It's your choice. Do you think it's purely coincidental that this this figure https://atlas.web.cern.ch/Atlas/GROUPS/PHYSICS/CONFNOTES/ATLAS-CONF-2012-170/fig_03.png https://atlas.web.cern.ch/Atlas/GROUPS/PHYSICS/CONFNOTES/ATL... includes the sigma=0 (null hypothesis) line at 0.5 and not at 1? (Hint: the expect value of the p-value under the null hypothesis is 0.5.) (That's a rhetorical question: I already know it's because this is not a well-formed experiment or something.)
    - jsprogrammer 11y ago
      >P-values are by definition uniformly distributed when the null hypothesis is true. Where are you getting this from? When the null hypothesis is true, the p-value should be 1. This follows directly from the definition. If the p-value is not 1 and the null hypothesis is in fact true, your experiment or calculations are wrong. You might also just have the wrong null hypothesis (eg. sensors have more noise than assumed).
      - jsprogrammer 11y ago
        I'm willing to accept that in some designs, the first observation of p-value may be uniformly distributed over (0,1], but, as additional observations are made, the value should converge to 0 or 1. What would be the purpose, or usefulness of p-value being uniformly distributed if the null-hypothesis is true? It's much simpler to design things to converge to a single number. Edit: I have also considered that p-value could be uniformly distributed if the null-hypothesis is false (where you claimed true). I don't know the answer to that.

kgwgk 11y ago

Ok, so you're thinking about a random variable which converges to some value when the null hypothesis is true. This is fine, but it has nothing to do whatsoever with p-values. Let me say that your notation is not very appropriate. It makes no sense to say that P(X|H) converges to 1. If you expect X to converge to C if the null hypothesis is true, you can simply say X->C. A proper notation involving probabilities would be P(|X-C|>epsilon)->0 for any positive epsilon (convergence in probability) or maybe P(X->C)=1 (convergence almost surely). Taking as you suggest X=(#tails/#heads), you expect that X->1 if the coin is fair (I'm not sure why you find this is not a well-defined null hypothesis, but I don't really care). However, P(X)<1 for every X. In fact, P(X=1)->0 as the number if trials increases (X will get closer to 1 on average, but getting exactly 1 will get more and more unlikely). As I said, you're free to prefer your converging statistics and your well-defined null hypothesis. But you should be aware that people are talking about something completely different when discussing things like the 1e-7 p-value in the Higgs boson discovery or the reproducibility of statistically significant results. EDIT: Another example, maybe better-defined: a random variable distributed (under the null hypothesis) x~Normal(mu=0,sigma=1). Let's say you take N samples (I let you choose the number, so I don't pick one which is not good enough).The statistic is the mean X=(x_1+x_2+..+x_N)/N. If the null hypothesis is true, X->mu=0. You get X=1/sqrt(N). What's your "p-value" in that case?

jsprogrammer 11y ago
>Ok, so you're thinking about a random variable which converges to some value when the null hypothesis is true. This is fine, but it has nothing to do whatsoever with p-values. Well, the variable itself doesn't, just the observed value of P(X|H). X can be any random variable, but typically it will need to be transformed to have a normal distribution about 0 with a standard deviation of 1 (since this is what the typical null-hypothesis predicts). To effectively use p-value analysis, it is typically assumed that your null-hypothesis predicts that your observations will be normally distributed with a mean of 0 and a standard deviation of 1. The total count of heads observed will not be distributed that way. Neither will the probability of a particular sequence (what your example seemed to be calculating). I say your null hypothesis is not well-defined because the term 'fair' remains undefined (though we could guess at the meaning) and in fact makes no predictions about the world. You need to apply transformations to your random variable so that it will appear normally distributed about 0 with a standard deviation of 1 if the hypothesis is true. >Let me say that your notation is not very appropriate. It makes no sense to say that P(X|H) converges to 1. My notation is perfectly appropriate. X is a random variable and a random variable is the only thing that can go there (if you are doing p-value analysis). X is not assumed to be uniform or simple (although it certainly could be). P(T>T(X)|H) can be replaced with P(Y|H) every time (Y = T>T(X)). >As I said, you're free to prefer your converging statistics and your well-defined null hypothesis. But you should be aware that people are talking about something completely different when discussing things like the 1e-7 p-value in the Higgs boson discovery or the reproducibility of statistically significant results. I'm glad that we finally agree on this (although I dispute that anyone working on the Higgs boson discovery disagrees with me). One of my first claims was that others may not be calculating true p-values, but may calculate something and call it 'p-value' and then think that it means something it does not. In fact, this entire topic even links to an article in a prominent publisher claiming the same. Do you think it is purely coincidental that the figures I showed you from the Higgs experiment show the lines converging towards only two different numbers: 1 and 0? Edit: You'll have to give me some time on your edit. It's not something I typically calculate and I have other business to attend to today.
- kgwgk 11y ago
  Ok, so maybe your definition does correspond to a p-value after all. It's hard to say as you have refused to discuss concrete cases (like the fair coin or the loaded die, which are standard examples to introduce p-values). But if you're actually calculating a p-value then it won't behave as you expect. It won't converge to anything (edit: if the null hypothesis holds). P-values are by definition uniformly distributed when the null hypothesis is true. If your "p-value" is not, then it's not a p-value. It really is that simple. Or maybe everyone else is using the wrong "p-values" and yours are the real thing. You can believe it if you want. Please disregard my previous questions, I see no point in continuing this discussion. But you might want to read a bit more about p-values: you won't find anyone (I hope!) sharing your point of view. Once you understand what the p-value is, and what it is not, you might indeed conclude that they are entirely useless. Of course it's your right to avoid learning what p-values really are, and keep the faith. It's your choice. Do you think it's purely coincidental that this this figure https://atlas.web.cern.ch/Atlas/GROUPS/PHYSICS/CONFNOTES/ATLAS-CONF-2012-170/fig_03.png https://atlas.web.cern.ch/Atlas/GROUPS/PHYSICS/CONFNOTES/ATL... includes the sigma=0 (null hypothesis) line at 0.5 and not at 1? (Hint: the expect value of the p-value under the null hypothesis is 0.5.) (That's a rhetorical question: I already know it's because this is not a well-formed experiment or something.)
  - jsprogrammer 11y ago
    >P-values are by definition uniformly distributed when the null hypothesis is true. Where are you getting this from? When the null hypothesis is true, the p-value should be 1. This follows directly from the definition. If the p-value is not 1 and the null hypothesis is in fact true, your experiment or calculations are wrong. You might also just have the wrong null hypothesis (eg. sensors have more noise than assumed).
    - jsprogrammer 11y ago
      I'm willing to accept that in some designs, the first observation of p-value may be uniformly distributed over (0,1], but, as additional observations are made, the value should converge to 0 or 1. What would be the purpose, or usefulness of p-value being uniformly distributed if the null-hypothesis is true? It's much simpler to design things to converge to a single number. Edit: I have also considered that p-value could be uniformly distributed if the null-hypothesis is false (where you claimed true). I don't know the answer to that.