6 ms·

...the inferential procedure of Bayesian statistics is to assume a prior distribution and a probability model for data and then use probability theory to determ

by mbil 7y ago

...the inferential procedure of Bayesian statistics is to assume a prior distribution and a probability model for data and then use probability theory to determine the posterior. But if these steps, or something approximating them, are necessary, if you can’t just look at your data and come up with a subjective posterior distribution, then how is it reasonable to suppose that you could able to come up with an unassailable subjective distribution before seeing the data? Is the point to end up with an "unassailable subjective distribution"? I believe the power of Bayesian thinking is that you can take a subjective prior, which is necessarily assailable in its subjectivity, and then combine it with data. The result is something that is better than either the subjective prior alone or the likelihood estimator gleaned from data alone.

skat20phys 7y ago
I haven't read the linked paper yet, but the blog post points a little to what's on my mind. To borrow a bit from a different paper and anonymous reviewer on one of my papers, inference serves different aims or philosophies. Sometimes it serves more of an estimation function, to increase information about some quantity, or to improve the estimate of that quantity. But sometimes it serves an evaluative, competitive function, in the Popperian sense of affording risky tests of one or more theories or models. In this latter Popperian aim of inference, priors are to be minimized, which is in many ways the opposite scenario that is assumed with standard subjective Bayesian methods. And even with the former "estimation" inferential aims, there may be situations where you truly have no information or don't feel comfortable assuming it. What's nice about Bayesian statistics is it still provides a framework for this scenario, in the form of reference priors, in that if nothing else your design and model supporting the parameter(s) to be inferred about implies some kind of assumptions about what you're making inferences about. That in turn can be transformed into a "least informative" prior. So it allows an objective Bayesian framework. However, in that framework, in many cases you're still often left with uniform priors, which then reduce to frequentist statistics. And in a broader sense, frequentist methods are even further removed from making assumptions in that they completely eliminate the prior from inferential consideration. There's a tension then, in that in small samples your priors will bias your estimates. If you use least informative priors, you're often doing something akin to frequentist methods anyway. And in large samples the likelihood dominates the posterior so it matters still less. From a certain perspective, ultimately with Bayesian methods you're making a bet that your priors are accurate enough that the increased bias in estimates will be small enough to be offset by decreased variance due to use of a prior. It's a gamble though, the risks of which will probably vary depending on the costs and benefits of different types of error. It's nice to see a paper trying to be honest about the problems with Bayesian inference, as it's a bit overhyped at the moment imho.
- pjcv 7y ago
  > However, in that framework, in many cases you're still often left with uniform priors, which then reduce to frequentist statistics. Several commenters have made this claim, which seems to be true only for maximum a posteriori (MAP) estimates. Frequentist methods do not construct a distribution over parameters.
  - skat20phys 7y ago
    That's a good point regarding MAP estimates versus, e.g., EAP estimates, although it does lead to questions about whether one or the other is preferable, and I think good arguments can be made either way.
zozbot234 7y ago
Yes, it's a bit ironic that the likelihood principle ("all the evidence drawn from data is summarized by the likelihood function") is something that both frequentist and Bayesian statisticians like to claim as their own. Frequentist statistics is basically Bayesian statistics done with a "flat" prior distribution (where "flat" can depend on how the model is parameterized, however). Seen from this POV, it may be somewhat weird to claim that Bayesian stats has "holes" in it.
- steerablesafe 7y ago
  Yes, the only difference is that in Bayesian stats the biggest hole (priors) is explicit and in your face. In frequentist stat that "flat" prior you mention is implicit and flies under the radar.
- gbrown 7y ago
  > Frequentist statistics is basically Bayesian statistics done with a "flat" prior distribution I disagree strongly - there’s a huge difference in the kind of inference which is provided by each paradigm. To oversimplify, frequentist statistics constructs decision rules with good properties concerning unknown (but fixed) parameters, while Bayesian statistics uses probability to directly reason about those parameters. Confidence and credible intervals are not the same thing - you’re actually flipping what’s considered fixed and random.
- 08-15 7y ago
  Maximum likelihood is equivalent to Bayesian maximum a-posteriori (MAP) given a uniform prior and the right choice of parameters. However, decision theory tells us, that MAP estimation is rarely, if ever, a good point estimator. The mean or the median of the posterior distribution is usually better, but the optimal choice may be something entirely different. (Why even consider MAP when it's dumb? Only because it's computationally cheaper than anything else.)
Retric 7y ago
Junk data can’t improve accuracy, that’s the flaw with Bayesian thinking. If I hand you a normal looking coin and ask what’s the odds for heads, tails, or edge to +\- 0.0001%, the absolute minimal trials is going to be based on zero priors. It’s valuable when you have a combination of useful prior knowledge and insufficient data to review. But, having useful prior knowledge can be difficult to distinguish from useless prior knowledge. Incorrect belief that something is safe means more people are harmed before you update that belief. This means you can’t say if Bayesian think is a net positive or a net negative abstractly.
- joshuamorton 7y ago
  I'm confused, in cases where you have truly no prior knowledge, such as your thought experiment, a bayesian can reasonably select a uniform prior. But often in real world statistical experiments, we have priors we can believe in.
  - Retric 7y ago
    You likely have fairly accurate priors about that coin flip I just mentioned. They are just not accurate enough to be useful. The math for designing sample sizes may be unfamiliar, so you can just try running the same simulation a few times. Then try a few priors on each data set and compare. This is effectively pure math so there is nothing to disagree or get upset about. What’s up for debate is how accurate your priors are in the real world.
    - nightski 7y ago
      Your comments apply to statistics and machine learning as a whole, not just Bayes statistics. At the end of the day all you have is your prior knowledge and what you learn from the data. If both of those are suspect, it does not matter what method you use. I just don't see how this can be considered a problem specific to Bayesian statistics.
      - Retric 7y ago

by mbil 7y ago

skat20phys 7y ago

I haven't read the linked paper yet, but the blog post points a little to what's on my mind. To borrow a bit from a different paper and anonymous reviewer on one of my papers, inference serves different aims or philosophies. Sometimes it serves more of an estimation function, to increase information about some quantity, or to improve the estimate of that quantity. But sometimes it serves an evaluative, competitive function, in the Popperian sense of affording risky tests of one or more theories or models. In this latter Popperian aim of inference, priors are to be minimized, which is in many ways the opposite scenario that is assumed with standard subjective Bayesian methods. And even with the former "estimation" inferential aims, there may be situations where you truly have no information or don't feel comfortable assuming it. What's nice about Bayesian statistics is it still provides a framework for this scenario, in the form of reference priors, in that if nothing else your design and model supporting the parameter(s) to be inferred about implies some kind of assumptions about what you're making inferences about. That in turn can be transformed into a "least informative" prior. So it allows an objective Bayesian framework. However, in that framework, in many cases you're still often left with uniform priors, which then reduce to frequentist statistics. And in a broader sense, frequentist methods are even further removed from making assumptions in that they completely eliminate the prior from inferential consideration. There's a tension then, in that in small samples your priors will bias your estimates. If you use least informative priors, you're often doing something akin to frequentist methods anyway. And in large samples the likelihood dominates the posterior so it matters still less. From a certain perspective, ultimately with Bayesian methods you're making a bet that your priors are accurate enough that the increased bias in estimates will be small enough to be offset by decreased variance due to use of a prior. It's a gamble though, the risks of which will probably vary depending on the costs and benefits of different types of error. It's nice to see a paper trying to be honest about the problems with Bayesian inference, as it's a bit overhyped at the moment imho.

pjcv 7y ago
> However, in that framework, in many cases you're still often left with uniform priors, which then reduce to frequentist statistics. Several commenters have made this claim, which seems to be true only for maximum a posteriori (MAP) estimates. Frequentist methods do not construct a distribution over parameters.
- skat20phys 7y ago
  That's a good point regarding MAP estimates versus, e.g., EAP estimates, although it does lead to questions about whether one or the other is preferable, and I think good arguments can be made either way.

zozbot234 7y ago

Yes, it's a bit ironic that the likelihood principle ("all the evidence drawn from data is summarized by the likelihood function") is something that both frequentist and Bayesian statisticians like to claim as their own. Frequentist statistics is basically Bayesian statistics done with a "flat" prior distribution (where "flat" can depend on how the model is parameterized, however). Seen from this POV, it may be somewhat weird to claim that Bayesian stats has "holes" in it.

steerablesafe 7y ago
Yes, the only difference is that in Bayesian stats the biggest hole (priors) is explicit and in your face. In frequentist stat that "flat" prior you mention is implicit and flies under the radar.
gbrown 7y ago
> Frequentist statistics is basically Bayesian statistics done with a "flat" prior distribution I disagree strongly - there’s a huge difference in the kind of inference which is provided by each paradigm. To oversimplify, frequentist statistics constructs decision rules with good properties concerning unknown (but fixed) parameters, while Bayesian statistics uses probability to directly reason about those parameters. Confidence and credible intervals are not the same thing - you’re actually flipping what’s considered fixed and random.
08-15 7y ago
Maximum likelihood is equivalent to Bayesian maximum a-posteriori (MAP) given a uniform prior and the right choice of parameters. However, decision theory tells us, that MAP estimation is rarely, if ever, a good point estimator. The mean or the median of the posterior distribution is usually better, but the optimal choice may be something entirely different. (Why even consider MAP when it's dumb? Only because it's computationally cheaper than anything else.)

Retric 7y ago

Junk data can’t improve accuracy, that’s the flaw with Bayesian thinking. If I hand you a normal looking coin and ask what’s the odds for heads, tails, or edge to +\- 0.0001%, the absolute minimal trials is going to be based on zero priors. It’s valuable when you have a combination of useful prior knowledge and insufficient data to review. But, having useful prior knowledge can be difficult to distinguish from useless prior knowledge. Incorrect belief that something is safe means more people are harmed before you update that belief. This means you can’t say if Bayesian think is a net positive or a net negative abstractly.

joshuamorton 7y ago
I'm confused, in cases where you have truly no prior knowledge, such as your thought experiment, a bayesian can reasonably select a uniform prior. But often in real world statistical experiments, we have priors we can believe in.
- Retric 7y ago
  You likely have fairly accurate priors about that coin flip I just mentioned. They are just not accurate enough to be useful. The math for designing sample sizes may be unfamiliar, so you can just try running the same simulation a few times. Then try a few priors on each data set and compare. This is effectively pure math so there is nothing to disagree or get upset about. What’s up for debate is how accurate your priors are in the real world.
  - nightski 7y ago
    Your comments apply to statistics and machine learning as a whole, not just Bayes statistics. At the end of the day all you have is your prior knowledge and what you learn from the data. If both of those are suspect, it does not matter what method you use. I just don't see how this can be considered a problem specific to Bayesian statistics.
    - Retric 7y ago