3 ms·

Honestly, I think Wasserman does a better job. The cookie interval example gets the fact that frequentists require uniform coverage properties with respect to t

Honestly, I think Wasserman does a better job. The cookie interval example gets the fact that frequentists require uniform coverage properties with respect to the unknown parameter right. But the "when you pull a cookie with 3 chips your interval is only correct 41% of the time" thing isn't really an essential Bayesian vs. Frequentist issue. As Wasserman notes, coverage is a minimal requirement for something being a confidence interval; we usually also construct them to avoid obvious deficiencies. All this objection shows is that the particular procedure in the example might not be the best one; it's not an argument that can be applied to all CIs in general. (And it's clear, if you play with the numbers, that the example can be improved.) Also, regarding "But note that we probably don't really care about the confidence interval or credibility interval." Many times we do - giving a point estimate and an associated quantification of its uncertainty is one of the most basic statistical tasks. Further, it's somewhat misleading to critique frequentists by saying they don't give probabilities for P(Jar | Chips), because in the frequentist setup the jar is a fixed and unknown parameter, not stochastic. For the two-cookie setting, it's trivial to generalize the construction in the M.SE post, so saying "frquentists can't track evidence to get better predictions" is simply wrong.

lalaithion 4y ago
"giving a point estimate and an associated quantification of its uncertainty is one of the most basic statistical tasks". My argument is that this is only true in a frequentist framing. A bayesian framing would ask, "why do you need a point estimate when you have the posterior?" In the cookie-jar case, what do you actually need to do in the real world that a confidence interval of jars helps you with? Do you win a prize if you guess the jar right? Do you win a prize if you guess the next cookie right? Do you need to select a strategy up front which gives at least a 70% chance of being right about the jar over iterations where the jar is fixed but the data varies? In the last case, a Bayesian directly computes a confidence interval. But in other cases, a confidence interval is just throwing away data for no reason. Frequentists are fundamentally unable to answer "Given that I drew a cookie with 2 chips on the first draw, what is the chance I draw a cookie with 0 chips on my second draw?". That question requires a prior; any solution frequentists find is just implicitly assigning a prior to the distribution of jars.
- spekcular 4y ago
  Well, I am not really interested in cookie jars. But I am interested in, for example, particle physics. There we need need simple ways to communicate point estimates and the associated uncertainties for various parameters of nature. Intervals are a convenient way to do this. Frequentist confidence intervals have the virtue that they will cover the true parameter at the nominal rate. Bayesian credible intervals in general have no such guarantee. In many cases we can find Bayesian-inspired interval estimators that have good coverage properties. But in some cases there is an irreconcilable conflict. And there you have to choose between long-run correctness and being Bayesian. You might claim that we should do away with intervals and just report posteriors for all physical quantities. This complicates matters slightly without solving the problem. If the true parameters, when ultimately known, consistently end up in very low density regions of the probability distribution (such as far in the tails), we would regard our uncertainty estimates as poor. Again, it is not hard to construct examples where Bayesian methods have poor coverage properties in this sense. (Also, a minor point: Regarding "Do you need to select a strategy up front which gives at least a 70% chance of being right about the jar over iterations where the jar is fixed but the data varies?", one does not need to fix the jar for frequentist methods to have good guarantees. See Wasserman's simulation with the median.) Re: "Given that I drew a cookie with 2 chips on the first draw, what is the chance I draw a cookie with 0 chips on my second draw?", this is not a question about estimating an unknown parameter of a distribution, so it's not statistical in the sense Wasserman is talking about. It's just an elementary probability question that requires knowing something about the jars to answer. Both frequentist and Bayesian statisticians agree on the validity of Bayes rule (and hence how to answer this question once the relevant information is known or assumed); where they differ is on how to conceptualize and estimate unknown parameters of probability distributions.
  - lalaithion 4y ago
    > I am interested in, for example, particle physics. There we need need simple ways to communicate point estimates and the associated uncertainties for various parameters of nature. Okay, but what do you _do_ with a confidence interval once you have it? It's just an abstract object that can't be used to take your knowledge and make better predictions about the future. If I tell you "This new particle decays with a half life of 28 years with a 95% confidence interval of +/- 5 years", can you take that information and use it to estimate the age of an object that started with 236 particles and now has 182 particles? > this is not a question about estimating an unknown parameter of a distribution, so it's not statistical in the sense Wasserman is talking about And a frequentist confidence interval doesn't answer a question about how you should update your knowledge so you can make better predictions in the future, so it's not statistical in the sense bayesians talk about.

lalaithion 4y ago

"giving a point estimate and an associated quantification of its uncertainty is one of the most basic statistical tasks". My argument is that this is only true in a frequentist framing. A bayesian framing would ask, "why do you need a point estimate when you have the posterior?" In the cookie-jar case, what do you actually need to do in the real world that a confidence interval of jars helps you with? Do you win a prize if you guess the jar right? Do you win a prize if you guess the next cookie right? Do you need to select a strategy up front which gives at least a 70% chance of being right about the jar over iterations where the jar is fixed but the data varies? In the last case, a Bayesian directly computes a confidence interval. But in other cases, a confidence interval is just throwing away data for no reason. Frequentists are fundamentally unable to answer "Given that I drew a cookie with 2 chips on the first draw, what is the chance I draw a cookie with 0 chips on my second draw?". That question requires a prior; any solution frequentists find is just implicitly assigning a prior to the distribution of jars.

spekcular 4y ago
Well, I am not really interested in cookie jars. But I am interested in, for example, particle physics. There we need need simple ways to communicate point estimates and the associated uncertainties for various parameters of nature. Intervals are a convenient way to do this. Frequentist confidence intervals have the virtue that they will cover the true parameter at the nominal rate. Bayesian credible intervals in general have no such guarantee. In many cases we can find Bayesian-inspired interval estimators that have good coverage properties. But in some cases there is an irreconcilable conflict. And there you have to choose between long-run correctness and being Bayesian. You might claim that we should do away with intervals and just report posteriors for all physical quantities. This complicates matters slightly without solving the problem. If the true parameters, when ultimately known, consistently end up in very low density regions of the probability distribution (such as far in the tails), we would regard our uncertainty estimates as poor. Again, it is not hard to construct examples where Bayesian methods have poor coverage properties in this sense. (Also, a minor point: Regarding "Do you need to select a strategy up front which gives at least a 70% chance of being right about the jar over iterations where the jar is fixed but the data varies?", one does not need to fix the jar for frequentist methods to have good guarantees. See Wasserman's simulation with the median.) Re: "Given that I drew a cookie with 2 chips on the first draw, what is the chance I draw a cookie with 0 chips on my second draw?", this is not a question about estimating an unknown parameter of a distribution, so it's not statistical in the sense Wasserman is talking about. It's just an elementary probability question that requires knowing something about the jars to answer. Both frequentist and Bayesian statisticians agree on the validity of Bayes rule (and hence how to answer this question once the relevant information is known or assumed); where they differ is on how to conceptualize and estimate unknown parameters of probability distributions.
- lalaithion 4y ago
  > I am interested in, for example, particle physics. There we need need simple ways to communicate point estimates and the associated uncertainties for various parameters of nature. Okay, but what do you _do_ with a confidence interval once you have it? It's just an abstract object that can't be used to take your knowledge and make better predictions about the future. If I tell you "This new particle decays with a half life of 28 years with a 95% confidence interval of +/- 5 years", can you take that information and use it to estimate the age of an object that started with 236 particles and now has 182 particles? > this is not a question about estimating an unknown parameter of a distribution, so it's not statistical in the sense Wasserman is talking about And a frequentist confidence interval doesn't answer a question about how you should update your knowledge so you can make better predictions in the future, so it's not statistical in the sense bayesians talk about.