5 ms·

Unless I've forgotten more than I hope, I believe the formal definition of a 95% confidence interval is that "if the model is true, 95% of the experiments would

by lutorm 8y ago

Unless I've forgotten more than I hope, I believe the formal definition of a 95% confidence interval is that "if the model is true, 95% of the experiments would result in a point estimate within the interval." This is distinctly different from "a 95% probability that the true value is contained within the confidence interval", but that is typically what is loosely inferred.

mr_toad 8y ago
Confidence intervals can be applied to point estimates, estimates of means, and estimates of other things, including higher order moments. The difference is hugely important, the central limit theorem and the implication of a normal distribution commonly applies to sample means. You can calculate confidence intervals for most (not all) other statistics, like point estimates, but the distributions might not be normal.
- mike_ivanov 8y ago
  "for most (not all)" - yes, if analytically. If you can afford bootstrapping, then it is just "for all".
BeetleB 8y ago
Nope. It is (from a frequentist's model of statistics), exactly what the article is claiming it isn't: If the model is true, and we repeat the experiment several times, 95% of the intervals we calculate will contain the true value. The actual CI you get in each experiment will differ. Another discrepancy between frequentists statistics and the article is that yes, the values at the boundary of your interval are as credible as in the center.
- ronjobber 8y ago
  This is the first post I've seen that states the definition of a CI correctly. Another note: 'confidence interval' typically refers to the frequentist meaning, whereas 'credibility interval' is used in the Bayesian setting, when describing an interval of the posterior with 95% probability (which is arguably more interpretable). The usages of the two terms do not seem to generally be strict, however.
- kgwgk 8y ago
  > the values at the boundary of your interval are as credible as in the center. What would that mean in the frequentist framework?
  - BeetleB 8y ago
    To be explicit, and using an example similar to the one in the article, if your CI is (2, 40), with the center being 21, there is no reason to believe that the true value is closer to 21 than to, say, 3. To provide an extreme case, during the Iraq war, epidemiologists did a survey and came up with an estimated number of deaths. The point value was 100K, and that's what all the newspapers ran with. But the actual journal paper had a CI of (8K, 194K). There's no reason to believe the true value is closer to 100K than it is to 10K. Or to 190K.
    - kgwgk 8y ago
      You're right, from the frequentist definition of a confidence interval (2,40) we can't say the the true value is more likely to be closer to 21 than to 3. But we can't neither say that the true value is equally likely to be closer to 21 than to 3. The point is that, from the frequentist definition of a confidence interval, there is nothing at all that we can say about how likely the true value is to be here or there. It could be 3, 21, or 666 and there is nothing that can be said about the likelihood of each value (unless we go beyond the frequentist framework and introduce prior probabilities).
      - BeetleB 8y ago
        >The point is that, from the frequentist definition of a confidence interval, there is nothing at all that we can say about how likely the true value is to be here or there. Yes - sorry if I wasn't clear. I did not mean to imply that each value in the interval is equally likely (and looking over my comments, I do not think I did imply that). The complaint is that the article is stating otherwise as fact. >One practical way to do so is to rename confidence intervals as ‘compatibility intervals’ >The point estimate is the most compatible, and values near it are more compatible than those near the limits. They simply are not in a frequentist model (which is the model most social scientists uses). I agree with the main thrust of the article in that there are many problems with P values. But I am surprised that a journal like Nature is allowing clearly problematic statements like these. I don't know enough about the Bayesian world to be able to state if his statement is wrong there as well, but if it is correct there, it is problematic that the authors did not state clearly that they are referring to the Bayesian model and not the frequentist one. (Not to get into a B vs F war here, but I remember a nice joke amongst statisticians. There are 2 types of statisticians: Those who practice Bayesian statistics, and those who practice both).
bunderbunder 8y ago
It is distinctly different, but I have to think really hard to follow the practical difference between "If we ran 100 more identical experiments, 95 of them would estimate a population mean result somewhere in this range", and "There's a 95% probability that the population mean falls somewhere in this range." The thing that makes it hurt is that what seems like a minor rewording of that statement, "We're 95% confident that the range we calculated contains the true population mean," would be correct. (It's not the definition of a CI, but it is implied by the definition of a CI.) And that the reason why one is true and the other is false is down to the technical distinction that the true population mean is a discrete value and not a draw from a random variable, so, from a strictly mathematical sense, it is nonsensical to apply probabilities to it. By a the same token, an even more subtle rewording gets us back to falsehood: "There's a 95% chance that the range we calculated contains the true population mean." Which, delving into to that level of hair-splitting makes for interesting math, but also leaves me with the opinion that, sure, the most common intuitive interpretation is wrong, but it's wrong in a way that isn't really of much practical importance. By contrast, the most common intuitive interpretation of the p-value is disastrously wrong.

by lutorm 8y ago

mr_toad 8y ago

Confidence intervals can be applied to point estimates, estimates of means, and estimates of other things, including higher order moments. The difference is hugely important, the central limit theorem and the implication of a normal distribution commonly applies to sample means. You can calculate confidence intervals for most (not all) other statistics, like point estimates, but the distributions might not be normal.

mike_ivanov 8y ago
"for most (not all)" - yes, if analytically. If you can afford bootstrapping, then it is just "for all".

BeetleB 8y ago

Nope. It is (from a frequentist's model of statistics), exactly what the article is claiming it isn't: If the model is true, and we repeat the experiment several times, 95% of the intervals we calculate will contain the true value. The actual CI you get in each experiment will differ. Another discrepancy between frequentists statistics and the article is that yes, the values at the boundary of your interval are as credible as in the center.

ronjobber 8y ago
This is the first post I've seen that states the definition of a CI correctly. Another note: 'confidence interval' typically refers to the frequentist meaning, whereas 'credibility interval' is used in the Bayesian setting, when describing an interval of the posterior with 95% probability (which is arguably more interpretable). The usages of the two terms do not seem to generally be strict, however.
kgwgk 8y ago
> the values at the boundary of your interval are as credible as in the center. What would that mean in the frequentist framework?
- BeetleB 8y ago
  To be explicit, and using an example similar to the one in the article, if your CI is (2, 40), with the center being 21, there is no reason to believe that the true value is closer to 21 than to, say, 3. To provide an extreme case, during the Iraq war, epidemiologists did a survey and came up with an estimated number of deaths. The point value was 100K, and that's what all the newspapers ran with. But the actual journal paper had a CI of (8K, 194K). There's no reason to believe the true value is closer to 100K than it is to 10K. Or to 190K.
  - kgwgk 8y ago
    You're right, from the frequentist definition of a confidence interval (2,40) we can't say the the true value is more likely to be closer to 21 than to 3. But we can't neither say that the true value is equally likely to be closer to 21 than to 3. The point is that, from the frequentist definition of a confidence interval, there is nothing at all that we can say about how likely the true value is to be here or there. It could be 3, 21, or 666 and there is nothing that can be said about the likelihood of each value (unless we go beyond the frequentist framework and introduce prior probabilities).
    - BeetleB 8y ago
      >The point is that, from the frequentist definition of a confidence interval, there is nothing at all that we can say about how likely the true value is to be here or there. Yes - sorry if I wasn't clear. I did not mean to imply that each value in the interval is equally likely (and looking over my comments, I do not think I did imply that). The complaint is that the article is stating otherwise as fact. >One practical way to do so is to rename confidence intervals as ‘compatibility intervals’ >The point estimate is the most compatible, and values near it are more compatible than those near the limits. They simply are not in a frequentist model (which is the model most social scientists uses). I agree with the main thrust of the article in that there are many problems with P values. But I am surprised that a journal like Nature is allowing clearly problematic statements like these. I don't know enough about the Bayesian world to be able to state if his statement is wrong there as well, but if it is correct there, it is problematic that the authors did not state clearly that they are referring to the Bayesian model and not the frequentist one. (Not to get into a B vs F war here, but I remember a nice joke amongst statisticians. There are 2 types of statisticians: Those who practice Bayesian statistics, and those who practice both).

bunderbunder 8y ago

It is distinctly different, but I have to think really hard to follow the practical difference between "If we ran 100 more identical experiments, 95 of them would estimate a population mean result somewhere in this range", and "There's a 95% probability that the population mean falls somewhere in this range." The thing that makes it hurt is that what seems like a minor rewording of that statement, "We're 95% confident that the range we calculated contains the true population mean," would be correct. (It's not the definition of a CI, but it is implied by the definition of a CI.) And that the reason why one is true and the other is false is down to the technical distinction that the true population mean is a discrete value and not a draw from a random variable, so, from a strictly mathematical sense, it is nonsensical to apply probabilities to it. By a the same token, an even more subtle rewording gets us back to falsehood: "There's a 95% chance that the range we calculated contains the true population mean." Which, delving into to that level of hair-splitting makes for interesting math, but also leaves me with the opinion that, sure, the most common intuitive interpretation is wrong, but it's wrong in a way that isn't really of much practical importance. By contrast, the most common intuitive interpretation of the p-value is disastrously wrong.