3 ms·

"giving a point estimate and an associated quantification of its uncertainty is one of the most basic statistical tasks". My argument is that this is only true

"giving a point estimate and an associated quantification of its uncertainty is one of the most basic statistical tasks". My argument is that this is only true in a frequentist framing. A bayesian framing would ask, "why do you need a point estimate when you have the posterior?" In the cookie-jar case, what do you actually need to do in the real world that a confidence interval of jars helps you with? Do you win a prize if you guess the jar right? Do you win a prize if you guess the next cookie right? Do you need to select a strategy up front which gives at least a 70% chance of being right about the jar over iterations where the jar is fixed but the data varies? In the last case, a Bayesian directly computes a confidence interval. But in other cases, a confidence interval is just throwing away data for no reason. Frequentists are fundamentally unable to answer "Given that I drew a cookie with 2 chips on the first draw, what is the chance I draw a cookie with 0 chips on my second draw?". That question requires a prior; any solution frequentists find is just implicitly assigning a prior to the distribution of jars.

spekcular 4y ago
Well, I am not really interested in cookie jars. But I am interested in, for example, particle physics. There we need need simple ways to communicate point estimates and the associated uncertainties for various parameters of nature. Intervals are a convenient way to do this. Frequentist confidence intervals have the virtue that they will cover the true parameter at the nominal rate. Bayesian credible intervals in general have no such guarantee. In many cases we can find Bayesian-inspired interval estimators that have good coverage properties. But in some cases there is an irreconcilable conflict. And there you have to choose between long-run correctness and being Bayesian. You might claim that we should do away with intervals and just report posteriors for all physical quantities. This complicates matters slightly without solving the problem. If the true parameters, when ultimately known, consistently end up in very low density regions of the probability distribution (such as far in the tails), we would regard our uncertainty estimates as poor. Again, it is not hard to construct examples where Bayesian methods have poor coverage properties in this sense. (Also, a minor point: Regarding "Do you need to select a strategy up front which gives at least a 70% chance of being right about the jar over iterations where the jar is fixed but the data varies?", one does not need to fix the jar for frequentist methods to have good guarantees. See Wasserman's simulation with the median.) Re: "Given that I drew a cookie with 2 chips on the first draw, what is the chance I draw a cookie with 0 chips on my second draw?", this is not a question about estimating an unknown parameter of a distribution, so it's not statistical in the sense Wasserman is talking about. It's just an elementary probability question that requires knowing something about the jars to answer. Both frequentist and Bayesian statisticians agree on the validity of Bayes rule (and hence how to answer this question once the relevant information is known or assumed); where they differ is on how to conceptualize and estimate unknown parameters of probability distributions.
- lalaithion 4y ago
  > I am interested in, for example, particle physics. There we need need simple ways to communicate point estimates and the associated uncertainties for various parameters of nature. Okay, but what do you _do_ with a confidence interval once you have it? It's just an abstract object that can't be used to take your knowledge and make better predictions about the future. If I tell you "This new particle decays with a half life of 28 years with a 95% confidence interval of +/- 5 years", can you take that information and use it to estimate the age of an object that started with 236 particles and now has 182 particles? > this is not a question about estimating an unknown parameter of a distribution, so it's not statistical in the sense Wasserman is talking about And a frequentist confidence interval doesn't answer a question about how you should update your knowledge so you can make better predictions in the future, so it's not statistical in the sense bayesians talk about.
  - spekcular 4y ago
    The blog post talks about inference, not prediction, so I find it odd you keep bringing up prediction tasks. There are interesting questions and differences here, but it is very much not the subject of the post. A standard frequentist tool for making predictions is the prediction interval. This is the appropriate comparison point for Bayesian prediction methods, and exactly the same issues arise as in the comparison of confidence intervals to credible intervals (or posteriors). Namely, frequentist prediction intervals have guaranteed error control, while Bayesian predictions generally do not. So in certain cases you have to choose between being right most of time about your predictions, and being Bayesian.

spekcular 4y ago

Well, I am not really interested in cookie jars. But I am interested in, for example, particle physics. There we need need simple ways to communicate point estimates and the associated uncertainties for various parameters of nature. Intervals are a convenient way to do this. Frequentist confidence intervals have the virtue that they will cover the true parameter at the nominal rate. Bayesian credible intervals in general have no such guarantee. In many cases we can find Bayesian-inspired interval estimators that have good coverage properties. But in some cases there is an irreconcilable conflict. And there you have to choose between long-run correctness and being Bayesian. You might claim that we should do away with intervals and just report posteriors for all physical quantities. This complicates matters slightly without solving the problem. If the true parameters, when ultimately known, consistently end up in very low density regions of the probability distribution (such as far in the tails), we would regard our uncertainty estimates as poor. Again, it is not hard to construct examples where Bayesian methods have poor coverage properties in this sense. (Also, a minor point: Regarding "Do you need to select a strategy up front which gives at least a 70% chance of being right about the jar over iterations where the jar is fixed but the data varies?", one does not need to fix the jar for frequentist methods to have good guarantees. See Wasserman's simulation with the median.) Re: "Given that I drew a cookie with 2 chips on the first draw, what is the chance I draw a cookie with 0 chips on my second draw?", this is not a question about estimating an unknown parameter of a distribution, so it's not statistical in the sense Wasserman is talking about. It's just an elementary probability question that requires knowing something about the jars to answer. Both frequentist and Bayesian statisticians agree on the validity of Bayes rule (and hence how to answer this question once the relevant information is known or assumed); where they differ is on how to conceptualize and estimate unknown parameters of probability distributions.

lalaithion 4y ago
> I am interested in, for example, particle physics. There we need need simple ways to communicate point estimates and the associated uncertainties for various parameters of nature. Okay, but what do you _do_ with a confidence interval once you have it? It's just an abstract object that can't be used to take your knowledge and make better predictions about the future. If I tell you "This new particle decays with a half life of 28 years with a 95% confidence interval of +/- 5 years", can you take that information and use it to estimate the age of an object that started with 236 particles and now has 182 particles? > this is not a question about estimating an unknown parameter of a distribution, so it's not statistical in the sense Wasserman is talking about And a frequentist confidence interval doesn't answer a question about how you should update your knowledge so you can make better predictions in the future, so it's not statistical in the sense bayesians talk about.
- spekcular 4y ago
  The blog post talks about inference, not prediction, so I find it odd you keep bringing up prediction tasks. There are interesting questions and differences here, but it is very much not the subject of the post. A standard frequentist tool for making predictions is the prediction interval. This is the appropriate comparison point for Bayesian prediction methods, and exactly the same issues arise as in the comparison of confidence intervals to credible intervals (or posteriors). Namely, frequentist prediction intervals have guaranteed error control, while Bayesian predictions generally do not. So in certain cases you have to choose between being right most of time about your predictions, and being Bayesian.