10 ms·

Is this not the definition of "confidence interval"? The first few Google results all define it this way...

by taktoa 8y ago

aidenn0 8y ago
That cannot be the result of an experiment without some sense of the prior probability, and I don't think CIs as suggested in the article account for that. For example, if I perform an experiment on a light source and find a 95% CI of 400-1200 lumens for the brightness, the actual probability of this being true is much higher if the light source is a 60W incandescent bulb than if the light source is the sun.
- mamon 8y ago
  except in real experimentation you control for all the variables except the one you're investigating. So it is not a very usefull thing to compare a bulb with the Sun. Instead, if you state that "luminescence OF THE 60W INCANDESCENT LIGHTBULB is 400-1200 lumens with 95% CI", then that's the usefull information that let's you set the right expectations when designing lightning for your new house, for example.
  - aidenn0 8y ago
    I think you missed the point of my example. I was suggesting that an experiment performed on the sun showing a 95% CI of the brightness being 400-1200 lumens should result in a reasonable person believing that the probability of the Sun's brightness falling in that range is approximately zero, while the same result for a 60W light-bulb should result in a reasonable person being more than 95% certain that the bulb's brightness falls in that range. Just like a large number of people misinterpret a P value of .01 to mean a 1% chance of the results being due to chance[1], CIs can be similarly misinterpreted. 1: A .01 P value actually means that if the null hypothesis is true, then you would get the result 1% of the time. The analogy to my above example would be that if I run an experiment and get a result that "the sun is less bright than a 60W light-bulb" with a P value of .01, it's almost certainly not true that the sun is less bright than the light-bulb, since the prior probability of the sun being less bright than a 60W light bulb is many orders of magnitude smaller than 1%.
khr 8y ago
A 95% confidence interval will contain the true mean 95% of the time (across an infinite number of replications of the experiment/study). For a single confidence interval, you have either captured the mean in your confidence interval, or you've not -- there's no probability about it.
- justinpombrio 8y ago
  I believe this is the correct frequentist interpretation. To quote wikipedia: > A 95% confidence level does not mean that for a given realized interval there is a 95% probability that the population parameter lies within the interval (i.e., a 95% probability that the interval covers the population parameter).[10] According to the strict frequentist interpretation, once an interval is calculated, this interval either covers the parameter value or it does not; it is no longer a matter of probability.
- EForEndeavour 8y ago
  This is where I get lost: > For a single confidence interval, you have either captured the mean in your confidence interval, or you've not -- there's no probability about it. Isn't there? The underlying truth is that you either definitely have or have not captured the population mean in any specific confidence interval. But you can't know this truth. In the long run, if "a 95% confidence interval contains the true mean 95% of the time across an infinite number of replications of the experiment/study," then isn't it true that any single specific experiment's CI has a 95% probability of containing the true value?! In my untrained mind, this is exactly equivalent to flipping an unfair coin with a 95% chance of heads. Sure, before flipping, the outcome of heads has a 95% probability. After flipping, you either get heads or tails. But if you flip a coin and hide the outcome without looking at it, doesn't it still have a 95% chance of being heads as far as the experimenter can tell?
dxbydt 8y ago
To see how absurd that definition is, think about this - CI itself is random! So if you conduct a 100 experiments, you'll get a 100 (non-overlapping) CI's! So in which of those 100 CI's does the "true population mean" lie ? All 100 of them ? 95 of them ?! You tell me.
- BeetleB 8y ago
  >So if you conduct a 100 experiments, you'll get a 100 (non-overlapping) CI's! So in which of those 100 CI's does the "true population mean" lie ? All 100 of them ? 95 of them ?! You tell me. I don't know why you get the idea that all 100 will be non-overlapping. That's simply false. And yes, if your assumptions were correct, regular (i.e. frequentist) statistics will state that roughly 95% of the CIs will contain the true mean. There is nothing absurd about it.
  - dxbydt 8y ago
    Speaking as a Stat TA, literally over 90% of the students taking the class will conduct 1 experiment, not 100, which gives you 1 CI, not 100, & then say that particular CI has a 95% chance of containing the population mean! Then when I tell them the mean is either in that CI or not ( either 100% in, or 0% in ), they google the CI definition & point me to that. That's why I said that definition doesn't work for the masses. It can be interpreted as - "if you conduct a 100 experiments & get 100 CI's then roughly 95 of those will contain true mean", but then nobody ever conducts 100 experiments, so from their pov its an absurd definition. Its imperative to understand that these definitions are written not for the average user of statistics, but for a trained statistician. Unfortunately, the average stat consumer vastly outnumbers the professional. Papers are littered with statements like p value proves H0, or proves H1. I have had numerous conversations with scientists ( not statisticians, but pharma/epidem/engg people who show up to the stat lab for consult ) that their p value doesn't prove H0 or H1. "What do you mean you can't prove H1 ? Oh you mean it only rejects H0 ? Ok but isn't that same as prove H1 ? It isn't ?! Well in my field if I just state it rejects H1 it won't be well understood so I am going to instead say H1 has been proved!" So there's little the statistician can do. Regards overlap, I meant total/exact overlap, as in no two CIs will be identical on any conti dist.
lutorm 8y ago
Unless I've forgotten more than I hope, I believe the formal definition of a 95% confidence interval is that "if the model is true, 95% of the experiments would result in a point estimate within the interval." This is distinctly different from "a 95% probability that the true value is contained within the confidence interval", but that is typically what is loosely inferred.
- mr_toad 8y ago
  Confidence intervals can be applied to point estimates, estimates of means, and estimates of other things, including higher order moments. The difference is hugely important, the central limit theorem and the implication of a normal distribution commonly applies to sample means. You can calculate confidence intervals for most (not all) other statistics, like point estimates, but the distributions might not be normal.
  - mike_ivanov 8y ago
    "for most (not all)" - yes, if analytically. If you can afford bootstrapping, then it is just "for all".
- BeetleB 8y ago
  Nope. It is (from a frequentist's model of statistics), exactly what the article is claiming it isn't: If the model is true, and we repeat the experiment several times, 95% of the intervals we calculate will contain the true value. The actual CI you get in each experiment will differ. Another discrepancy between frequentists statistics and the article is that yes, the values at the boundary of your interval are as credible as in the center.
  - ronjobber 8y ago
    This is the first post I've seen that states the definition of a CI correctly. Another note: 'confidence interval' typically refers to the frequentist meaning, whereas 'credibility interval' is used in the Bayesian setting, when describing an interval of the posterior with 95% probability (which is arguably more interpretable). The usages of the two terms do not seem to generally be strict, however.
pdkl95 8y ago
http://jakevdp.github.io/blog/2014/06/12/frequentism-and-bayesianism-3-confidence-credibility/ http://jakevdp.github.io/blog/2014/06/12/frequentism-and-bay...
- lucienlecam 8y ago
  This one of many Bayesian vs. frequentist blog posts where the frequentist example is presented in such bad faith or is so wrong that it's impossible to take seriously. Why is the sample mean used for the frequentist CI when it is not a sufficient statistic, and especially since it appears after the section discussing a "common sense approach" in which the author does mention a sufficient statistic: min(D)? All this blog post shows is reasonable bayesian approaches are better than frequentist approaches where common sense isn't allowed.
  - fenomas 8y ago
    Unless I'm missing something the author answers that here: > Edit, November 2014: ... Had we used, say, the Maximum Likelihood estimator or a sufficient estimator like min(x), our initial misinterpretation of the confidence interval would not have been as obviously wrong, and may even have fooled us into thinking we were right. But this does not change our central argument, which involves the question frequentism asks. Regardless of the estimator, if we try to use frequentism to ask about parameter values given observed data, we are making a mistake.
    - lucienlecam 8y ago
      In other words, the author has no mathematical examples to support the argument, and the objection is purely philosophical...
      - fenomas 8y ago
        Not as I understand it. Note that the author's argument there isn't "frequentism is bad because it gives an unreasonable answer here", it's "the fact that frequentism gives a different answer here demonstrates that it really is answering a different question".
alexgmcm 8y ago
Technically no for the standard frequentist confidence intervals, but if they use the Bayesian Credible Interval then I believe that would be the correct interpretation.

Is this not the definition of "confidence interval"? The first few Google results all define it this way...

by taktoa 8y ago

Is this not the definition of "confidence interval"? The first few Google results all define it this way...

That cannot be the result of an experiment without some sense of the prior probability, and I don't think CIs as suggested in the article account for that. For example, if I perform an experiment on a light source and find a 95% CI of 400-1200 lumens for the brightness, the actual probability of this being true is much higher if the light source is a 60W incandescent bulb than if the light source is the sun.

mamon 8y ago
except in real experimentation you control for all the variables except the one you're investigating. So it is not a very usefull thing to compare a bulb with the Sun. Instead, if you state that "luminescence OF THE 60W INCANDESCENT LIGHTBULB is 400-1200 lumens with 95% CI", then that's the usefull information that let's you set the right expectations when designing lightning for your new house, for example.
- aidenn0 8y ago
  I think you missed the point of my example. I was suggesting that an experiment performed on the sun showing a 95% CI of the brightness being 400-1200 lumens should result in a reasonable person believing that the probability of the Sun's brightness falling in that range is approximately zero, while the same result for a 60W light-bulb should result in a reasonable person being more than 95% certain that the bulb's brightness falls in that range. Just like a large number of people misinterpret a P value of .01 to mean a 1% chance of the results being due to chance[1], CIs can be similarly misinterpreted. 1: A .01 P value actually means that if the null hypothesis is true, then you would get the result 1% of the time. The analogy to my above example would be that if I run an experiment and get a result that "the sun is less bright than a 60W light-bulb" with a P value of .01, it's almost certainly not true that the sun is less bright than the light-bulb, since the prior probability of the sun being less bright than a 60W light bulb is many orders of magnitude smaller than 1%.

khr 8y ago

A 95% confidence interval will contain the true mean 95% of the time (across an infinite number of replications of the experiment/study). For a single confidence interval, you have either captured the mean in your confidence interval, or you've not -- there's no probability about it.

justinpombrio 8y ago
I believe this is the correct frequentist interpretation. To quote wikipedia: > A 95% confidence level does not mean that for a given realized interval there is a 95% probability that the population parameter lies within the interval (i.e., a 95% probability that the interval covers the population parameter).[10] According to the strict frequentist interpretation, once an interval is calculated, this interval either covers the parameter value or it does not; it is no longer a matter of probability.
EForEndeavour 8y ago
This is where I get lost: > For a single confidence interval, you have either captured the mean in your confidence interval, or you've not -- there's no probability about it. Isn't there? The underlying truth is that you either definitely have or have not captured the population mean in any specific confidence interval. But you can't know this truth. In the long run, if "a 95% confidence interval contains the true mean 95% of the time across an infinite number of replications of the experiment/study," then isn't it true that any single specific experiment's CI has a 95% probability of containing the true value?! In my untrained mind, this is exactly equivalent to flipping an unfair coin with a 95% chance of heads. Sure, before flipping, the outcome of heads has a 95% probability. After flipping, you either get heads or tails. But if you flip a coin and hide the outcome without looking at it, doesn't it still have a 95% chance of being heads as far as the experimenter can tell?

dxbydt 8y ago

To see how absurd that definition is, think about this - CI itself is random! So if you conduct a 100 experiments, you'll get a 100 (non-overlapping) CI's! So in which of those 100 CI's does the "true population mean" lie ? All 100 of them ? 95 of them ?! You tell me.

BeetleB 8y ago
>So if you conduct a 100 experiments, you'll get a 100 (non-overlapping) CI's! So in which of those 100 CI's does the "true population mean" lie ? All 100 of them ? 95 of them ?! You tell me. I don't know why you get the idea that all 100 will be non-overlapping. That's simply false. And yes, if your assumptions were correct, regular (i.e. frequentist) statistics will state that roughly 95% of the CIs will contain the true mean. There is nothing absurd about it.
- dxbydt 8y ago
  Speaking as a Stat TA, literally over 90% of the students taking the class will conduct 1 experiment, not 100, which gives you 1 CI, not 100, & then say that particular CI has a 95% chance of containing the population mean! Then when I tell them the mean is either in that CI or not ( either 100% in, or 0% in ), they google the CI definition & point me to that. That's why I said that definition doesn't work for the masses. It can be interpreted as - "if you conduct a 100 experiments & get 100 CI's then roughly 95 of those will contain true mean", but then nobody ever conducts 100 experiments, so from their pov its an absurd definition. Its imperative to understand that these definitions are written not for the average user of statistics, but for a trained statistician. Unfortunately, the average stat consumer vastly outnumbers the professional. Papers are littered with statements like p value proves H0, or proves H1. I have had numerous conversations with scientists ( not statisticians, but pharma/epidem/engg people who show up to the stat lab for consult ) that their p value doesn't prove H0 or H1. "What do you mean you can't prove H1 ? Oh you mean it only rejects H0 ? Ok but isn't that same as prove H1 ? It isn't ?! Well in my field if I just state it rejects H1 it won't be well understood so I am going to instead say H1 has been proved!" So there's little the statistician can do. Regards overlap, I meant total/exact overlap, as in no two CIs will be identical on any conti dist.

lutorm 8y ago

Unless I've forgotten more than I hope, I believe the formal definition of a 95% confidence interval is that "if the model is true, 95% of the experiments would result in a point estimate within the interval." This is distinctly different from "a 95% probability that the true value is contained within the confidence interval", but that is typically what is loosely inferred.

mr_toad 8y ago
Confidence intervals can be applied to point estimates, estimates of means, and estimates of other things, including higher order moments. The difference is hugely important, the central limit theorem and the implication of a normal distribution commonly applies to sample means. You can calculate confidence intervals for most (not all) other statistics, like point estimates, but the distributions might not be normal.
- mike_ivanov 8y ago
  "for most (not all)" - yes, if analytically. If you can afford bootstrapping, then it is just "for all".
BeetleB 8y ago
Nope. It is (from a frequentist's model of statistics), exactly what the article is claiming it isn't: If the model is true, and we repeat the experiment several times, 95% of the intervals we calculate will contain the true value. The actual CI you get in each experiment will differ. Another discrepancy between frequentists statistics and the article is that yes, the values at the boundary of your interval are as credible as in the center.
- ronjobber 8y ago
  This is the first post I've seen that states the definition of a CI correctly. Another note: 'confidence interval' typically refers to the frequentist meaning, whereas 'credibility interval' is used in the Bayesian setting, when describing an interval of the posterior with 95% probability (which is arguably more interpretable). The usages of the two terms do not seem to generally be strict, however.

pdkl95 8y ago

http://jakevdp.github.io/blog/2014/06/12/frequentism-and-bayesianism-3-confidence-credibility/ http://jakevdp.github.io/blog/2014/06/12/frequentism-and-bay...

lucienlecam 8y ago
This one of many Bayesian vs. frequentist blog posts where the frequentist example is presented in such bad faith or is so wrong that it's impossible to take seriously. Why is the sample mean used for the frequentist CI when it is not a sufficient statistic, and especially since it appears after the section discussing a "common sense approach" in which the author does mention a sufficient statistic: min(D)? All this blog post shows is reasonable bayesian approaches are better than frequentist approaches where common sense isn't allowed.
- fenomas 8y ago
  Unless I'm missing something the author answers that here: > Edit, November 2014: ... Had we used, say, the Maximum Likelihood estimator or a sufficient estimator like min(x), our initial misinterpretation of the confidence interval would not have been as obviously wrong, and may even have fooled us into thinking we were right. But this does not change our central argument, which involves the question frequentism asks. Regardless of the estimator, if we try to use frequentism to ask about parameter values given observed data, we are making a mistake.
  - lucienlecam 8y ago
    In other words, the author has no mathematical examples to support the argument, and the objection is purely philosophical...
    - fenomas 8y ago
      Not as I understand it. Note that the author's argument there isn't "frequentism is bad because it gives an unreasonable answer here", it's "the fact that frequentism gives a different answer here demonstrates that it really is answering a different question".

alexgmcm 8y ago

Technically no for the standard frequentist confidence intervals, but if they use the Bayesian Credible Interval then I believe that would be the correct interpretation.