4 ms·

> Because what the p-value is really telling you is the probability that on repeating the experiment you would see a result as or more extreme than what you obs

by Hermel 11y ago

> Because what the p-value is really telling you is the probability that on repeating the experiment you would see a result as or more extreme than what you observed. That's incorrect. If you perform the exact same experiment twice, the chances are exactly 50% that the first result is more extreme than the second one (neglecting equal outcomes). This illustrates nicely how hard it is to properly explain the p-value. Things would be more intuitive if people would report confidence intervals. I prefer "x is with a likelihood of 95% between a and b" a thousand times over "we measured x=c and reject the null hypothesis with 95% probability".

thinkmoore 11y ago
No, my description is correct, though I left out "assuming the null hypothesis is true." The equal probability you are describing is for the p-value itself being more extreme, not the result.
- Hermel 11y ago
  Yes, adding the clause with the null hypothesis corrects your statement.
sharnett 11y ago
"x is with a likelihood of 95% between a and b" -- that is not what a confidence interval means. Confidence intervals are about as confusing as p-values and don't really solve the problem.
- Retra 11y ago
  Statistics are just plain confusing all around.
gjm11 11y ago
> the chances are exactly 50% that the first result is more extreme than the second one The chance is 50% if you condition only on the information you have before doing either experiment. But once you've done the first experiment, the chance of a more extreme result given what happened the first time may be much more or much less. (Extreme example: Your experiment consists of rolling ten ordinary 6-sided dice. The null hypothesis is that they're fair dice, fairly rolled, in which case you expect a total not too different from 35 pips. All the dice come up 6. It is not now true that if you run the experiment again, you're as likely to get a more extreme result as you are to get a less extreme one!) > confidence intervals [...] "x is with a likelihood of 95% between a and b" But that isn't what a confidence interval means! A 95% confidence interval [a,b] means "If we ran the experiment lots of times, using the same method of computing the interval [a,b] each time, then in 95% of runs (in the long run) the true value would be in the interval [a,b] obtained on that run". (What you described is what Bayesians call a "credible interval". Of course that interval depends on your prior.)
- deleted 11y ago
  [deleted]
- Hermel 11y ago
  > in 95% of runs (in the long run) the true value would be in the interval [a,b] obtained on that run So 95% of the runs produce an interval that includes the true value. Doesn't that imply that when doing one run, there is a 95% chance of the resulting interval including the true value? I.e. the probability is 95% that the true value is in the interval?
  - kgwgk 11y ago
    Not really. For example, following a procedure which behaves well in the long run and provides 95% coverage you could get a confidence interval that contains only impossible values. Consider also the following example: you want to estimate x from two random numbers taken uniformly from the interval[x-1,x+1]. The interval defined by the two random numbers is a 50% confidence interval (half of the times x will be in the middle of the two). However, for some specific realizations (when the distance between the numbers is grater than one) you will be able to say with certainty that the interval contains x. See also https://github.com/richarddmorey/ConfidenceIntervalsFallacy https://github.com/richarddmorey/ConfidenceIntervalsFallacy
    - Hermel 11y ago
      Thanks, you are right. I thought I understood confidence intervals, but apparently I did not.
  - gjm11 11y ago
    kgwgk's explained why the answer is no, but let me offer a slightly different perspective. The situation is similar to the one we discussed earlier where you do the same experiment twice. Before you do the experiment, your estimate of Pr(true value is in confidence interval) is 95%. But after you do the experiment, you have extra information: you know how the experiment turned out. And this may lead you to a different estimate of Pr(true value is in the confidence interval).

by Hermel 11y ago

thinkmoore 11y ago

No, my description is correct, though I left out "assuming the null hypothesis is true." The equal probability you are describing is for the p-value itself being more extreme, not the result.

Hermel 11y ago
Yes, adding the clause with the null hypothesis corrects your statement.

sharnett 11y ago

"x is with a likelihood of 95% between a and b" -- that is not what a confidence interval means. Confidence intervals are about as confusing as p-values and don't really solve the problem.

Retra 11y ago
Statistics are just plain confusing all around.

gjm11 11y ago

> the chances are exactly 50% that the first result is more extreme than the second one The chance is 50% if you condition only on the information you have before doing either experiment. But once you've done the first experiment, the chance of a more extreme result given what happened the first time may be much more or much less. (Extreme example: Your experiment consists of rolling ten ordinary 6-sided dice. The null hypothesis is that they're fair dice, fairly rolled, in which case you expect a total not too different from 35 pips. All the dice come up 6. It is not now true that if you run the experiment again, you're as likely to get a more extreme result as you are to get a less extreme one!) > confidence intervals [...] "x is with a likelihood of 95% between a and b" But that isn't what a confidence interval means! A 95% confidence interval [a,b] means "If we ran the experiment lots of times, using the same method of computing the interval [a,b] each time, then in 95% of runs (in the long run) the true value would be in the interval [a,b] obtained on that run". (What you described is what Bayesians call a "credible interval". Of course that interval depends on your prior.)

deleted 11y ago
[deleted]
Hermel 11y ago
> in 95% of runs (in the long run) the true value would be in the interval [a,b] obtained on that run So 95% of the runs produce an interval that includes the true value. Doesn't that imply that when doing one run, there is a 95% chance of the resulting interval including the true value? I.e. the probability is 95% that the true value is in the interval?
- kgwgk 11y ago
  Not really. For example, following a procedure which behaves well in the long run and provides 95% coverage you could get a confidence interval that contains only impossible values. Consider also the following example: you want to estimate x from two random numbers taken uniformly from the interval[x-1,x+1]. The interval defined by the two random numbers is a 50% confidence interval (half of the times x will be in the middle of the two). However, for some specific realizations (when the distance between the numbers is grater than one) you will be able to say with certainty that the interval contains x. See also https://github.com/richarddmorey/ConfidenceIntervalsFallacy https://github.com/richarddmorey/ConfidenceIntervalsFallacy
  - Hermel 11y ago
    Thanks, you are right. I thought I understood confidence intervals, but apparently I did not.
- gjm11 11y ago
  kgwgk's explained why the answer is no, but let me offer a slightly different perspective. The situation is similar to the one we discussed earlier where you do the same experiment twice. Before you do the experiment, your estimate of Pr(true value is in confidence interval) is 95%. But after you do the experiment, you have extra information: you know how the experiment turned out. And this may lead you to a different estimate of Pr(true value is in the confidence interval).