4 ms·

This falls apart in higher dimensions, but in the example given in the article the two answers only differ because they have different priors. If you repeat the

This falls apart in higher dimensions, but in the example given in the article the two answers only differ because they have different priors. If you repeat the bayesian analysis using the prior \theta ~ N(0, x), and let x go to infinity, then you approach the frequentist answer. In my opinion, https://stats.stackexchange.com/questions/2272/whats-the-difference-between-a-confidence-interval-and-a-credible-interval https://stats.stackexchange.com/questions/2272/whats-the-dif... is a better explanation of the difference between confidence and credible intervals. Editing to add more commentary to my link: If you've read the link, one of the principal objections of the frequentist is "What if the jar is type B? Then your interval will be wrong 80% of the time, and only correct 20% of the time!" This is because, if you look at the original numbers, jar B has all types of cookies, and therefore any single draw from jar B can "look like" any other jar, and because other jars have more concentrated cookie types, they are "more likely" answers for each potential sample. This issue also comes up with the frequentist analysis! If you look at the confidence intervals, they all say "This jar could be jar B". Instead of being bad at detecting jar B, they are good at always considering jar B no matter the evidence – but it's the same uncertainty. The bayesian version of this objection is "when you pull a cookie with 3 chips your interval is only correct 41% of the time". This is because if we get jar A, we'll probably draw a 2-chip cookie, so it's outside of our confidence interval. But note that we probably don't really care about the confidence interval or credibility interval. It's basically a hack to take a probabilistic problem and turn our answer into black and white. To say "these hypotheses are valid and these are invalid". But this is statistics! If you just take the Bayesian approach, and throw out the need to create an arbitrary interval, you can just stop at the table titled P(Jar|Chips). That's all the information you need. If you draw a N-chip cookie, you can use that table to update your P(Chips_2) for a second draw, and you'll get a concrete probabilistic answer. Yes, you have to assume a prior. But frequentist statistics literally can't answer this question! Without a prior, there's no way to turn P(Chips | Jar) into a P(Jar | Chips) to update on, so you can't track your evidence to get better predictions. You just sit there saying "well, my interval meets the criterion even in the worst case".

spekcular 4y ago
Honestly, I think Wasserman does a better job. The cookie interval example gets the fact that frequentists require uniform coverage properties with respect to the unknown parameter right. But the "when you pull a cookie with 3 chips your interval is only correct 41% of the time" thing isn't really an essential Bayesian vs. Frequentist issue. As Wasserman notes, coverage is a minimal requirement for something being a confidence interval; we usually also construct them to avoid obvious deficiencies. All this objection shows is that the particular procedure in the example might not be the best one; it's not an argument that can be applied to all CIs in general. (And it's clear, if you play with the numbers, that the example can be improved.) Also, regarding "But note that we probably don't really care about the confidence interval or credibility interval." Many times we do - giving a point estimate and an associated quantification of its uncertainty is one of the most basic statistical tasks. Further, it's somewhat misleading to critique frequentists by saying they don't give probabilities for P(Jar | Chips), because in the frequentist setup the jar is a fixed and unknown parameter, not stochastic. For the two-cookie setting, it's trivial to generalize the construction in the M.SE post, so saying "frquentists can't track evidence to get better predictions" is simply wrong.
- lalaithion 4y ago
  "giving a point estimate and an associated quantification of its uncertainty is one of the most basic statistical tasks". My argument is that this is only true in a frequentist framing. A bayesian framing would ask, "why do you need a point estimate when you have the posterior?" In the cookie-jar case, what do you actually need to do in the real world that a confidence interval of jars helps you with? Do you win a prize if you guess the jar right? Do you win a prize if you guess the next cookie right? Do you need to select a strategy up front which gives at least a 70% chance of being right about the jar over iterations where the jar is fixed but the data varies? In the last case, a Bayesian directly computes a confidence interval. But in other cases, a confidence interval is just throwing away data for no reason. Frequentists are fundamentally unable to answer "Given that I drew a cookie with 2 chips on the first draw, what is the chance I draw a cookie with 0 chips on my second draw?". That question requires a prior; any solution frequentists find is just implicitly assigning a prior to the distribution of jars.

spekcular 4y ago

Honestly, I think Wasserman does a better job. The cookie interval example gets the fact that frequentists require uniform coverage properties with respect to the unknown parameter right. But the "when you pull a cookie with 3 chips your interval is only correct 41% of the time" thing isn't really an essential Bayesian vs. Frequentist issue. As Wasserman notes, coverage is a minimal requirement for something being a confidence interval; we usually also construct them to avoid obvious deficiencies. All this objection shows is that the particular procedure in the example might not be the best one; it's not an argument that can be applied to all CIs in general. (And it's clear, if you play with the numbers, that the example can be improved.) Also, regarding "But note that we probably don't really care about the confidence interval or credibility interval." Many times we do - giving a point estimate and an associated quantification of its uncertainty is one of the most basic statistical tasks. Further, it's somewhat misleading to critique frequentists by saying they don't give probabilities for P(Jar | Chips), because in the frequentist setup the jar is a fixed and unknown parameter, not stochastic. For the two-cookie setting, it's trivial to generalize the construction in the M.SE post, so saying "frquentists can't track evidence to get better predictions" is simply wrong.

lalaithion 4y ago
"giving a point estimate and an associated quantification of its uncertainty is one of the most basic statistical tasks". My argument is that this is only true in a frequentist framing. A bayesian framing would ask, "why do you need a point estimate when you have the posterior?" In the cookie-jar case, what do you actually need to do in the real world that a confidence interval of jars helps you with? Do you win a prize if you guess the jar right? Do you win a prize if you guess the next cookie right? Do you need to select a strategy up front which gives at least a 70% chance of being right about the jar over iterations where the jar is fixed but the data varies? In the last case, a Bayesian directly computes a confidence interval. But in other cases, a confidence interval is just throwing away data for no reason. Frequentists are fundamentally unable to answer "Given that I drew a cookie with 2 chips on the first draw, what is the chance I draw a cookie with 0 chips on my second draw?". That question requires a prior; any solution frequentists find is just implicitly assigning a prior to the distribution of jars.