3 ms·

The choice is not between using or not using priors, but between admitting it or denying it. A calculation that "does not use priors" is usually mathematically

The choice is not between using or not using priors, but between admitting it or denying it. A calculation that "does not use priors" is usually mathematically equivalent to one that does, for some specific prior.

therobots927 3y ago
The central limit theorem lets us calculate confidence intervals around population statistics regardless of the underlying distribution. If you're talking about working with small datasets, or speculating about the probability of events that have never occurred (n = 0), then just admit that math does not apply and that you're speculating.
- kgwgk 3y ago
  The problem with frequentist confidence intervals is that they don't really tell us anything about the probability that the statistic of interest for our population is contained in the interval - regardless of the size of our dataset. Not everyone wants to admit - or even understands - that the % confidence is about how frequently such intervals will contain the true value when the procedure is applied to other populations / datasets.
  - therobots927 3y ago
    Could you explain what the difference is between the two following segments in your comment: "[frequentist confidence intervals] don't really tell us anything about the probability that the statistic of interest for our population is contained in the interval" and "the % confidence is about how frequently such intervals will contain the true value when the procedure is applied to other populations / datasets" My understanding is that a 95% confidence interval implies that there is a 95% chance the true statistic lies within the interval. What are you saying it means? To some degree I'm interested in the philosophy behind probability but there are limits to my concern with it. If you're saying that ontologically this is not truly a 95% confidence interval, but practically speaking, it does mean that, then I'm more interested in the practical interpretation because it's more relevant to applications of statistics.
    - kgwgk 3y ago
      > My understanding is that a 95% confidence interval implies that there is a 95% chance the true statistic lies within the interval. What are you saying it means? If you calculate one thousand confidence intervals in 950 the true statistic will lie within the interval. I'm not sure if that means that there is a 95% chance the true statistic lies within a given interval. Let me use a extreme example to highlight the issue. Imagine that I use the following procedure to produce a 95% confidence interval for whatever quantity you're interested in: with 95% probability return the interval [ -1e999 1e999 ] with 5% probability return the interval [ -1e999 -1e999+1 ] If you calculate one thousand confidence intervals in 950 the true statistic will lie within the interval. I wouldn't say in any given case that there is a 95% chance the true statistic lies within that particular interval. (There was a 95% chance though.) This article (open access PDF available) discusses how different methods of producing confidence intervals perform in a toy problem: The fallacy of placing confidence in confidence intervals https://link.springer.com/article/10.3758/s13423-015-0947-8 https://link.springer.com/article/10.3758/s13423-015-0947-8 One could argue that those misleading flaws can be avoided by being careful with what confidence intervals do you use and how - but the point is that coverage probabilities are about a set of counterfactual situations and not about the actual situation at hand.
      - therobots927 3y ago
        I explicitely said I'm mostly interested in practical applications, and your example has e999 in it. Thank you for the PDF. I'll take a look at it, but it's really not helping the Bayesian case to start with an example like that. Is the assumption that the interval has a 95% chance of containing the true statistic going to lead to a bridge collapsing or an incorrect financial projection or not?
        kgwgk 3y ago
        Well, you asked for a clarification about the difference. Of course you're free to not care.

therobots927 3y ago

The central limit theorem lets us calculate confidence intervals around population statistics regardless of the underlying distribution. If you're talking about working with small datasets, or speculating about the probability of events that have never occurred (n = 0), then just admit that math does not apply and that you're speculating.

kgwgk 3y ago
The problem with frequentist confidence intervals is that they don't really tell us anything about the probability that the statistic of interest for our population is contained in the interval - regardless of the size of our dataset. Not everyone wants to admit - or even understands - that the % confidence is about how frequently such intervals will contain the true value when the procedure is applied to other populations / datasets.
- therobots927 3y ago
  Could you explain what the difference is between the two following segments in your comment: "[frequentist confidence intervals] don't really tell us anything about the probability that the statistic of interest for our population is contained in the interval" and "the % confidence is about how frequently such intervals will contain the true value when the procedure is applied to other populations / datasets" My understanding is that a 95% confidence interval implies that there is a 95% chance the true statistic lies within the interval. What are you saying it means? To some degree I'm interested in the philosophy behind probability but there are limits to my concern with it. If you're saying that ontologically this is not truly a 95% confidence interval, but practically speaking, it does mean that, then I'm more interested in the practical interpretation because it's more relevant to applications of statistics.
  - kgwgk 3y ago
    > My understanding is that a 95% confidence interval implies that there is a 95% chance the true statistic lies within the interval. What are you saying it means? If you calculate one thousand confidence intervals in 950 the true statistic will lie within the interval. I'm not sure if that means that there is a 95% chance the true statistic lies within a given interval. Let me use a extreme example to highlight the issue. Imagine that I use the following procedure to produce a 95% confidence interval for whatever quantity you're interested in: with 95% probability return the interval [ -1e999 1e999 ] with 5% probability return the interval [ -1e999 -1e999+1 ] If you calculate one thousand confidence intervals in 950 the true statistic will lie within the interval. I wouldn't say in any given case that there is a 95% chance the true statistic lies within that particular interval. (There was a 95% chance though.) This article (open access PDF available) discusses how different methods of producing confidence intervals perform in a toy problem: The fallacy of placing confidence in confidence intervals https://link.springer.com/article/10.3758/s13423-015-0947-8 https://link.springer.com/article/10.3758/s13423-015-0947-8 One could argue that those misleading flaws can be avoided by being careful with what confidence intervals do you use and how - but the point is that coverage probabilities are about a set of counterfactual situations and not about the actual situation at hand.
    - therobots927 3y ago
      I explicitely said I'm mostly interested in practical applications, and your example has e999 in it. Thank you for the PDF. I'll take a look at it, but it's really not helping the Bayesian case to start with an example like that. Is the assumption that the interval has a 95% chance of containing the true statistic going to lead to a bridge collapsing or an incorrect financial projection or not?
      - kgwgk 3y ago
        Well, you asked for a clarification about the difference. Of course you're free to not care.