5 ms·

This is a great book. However, I really hate the "Golem of Prague" introduction. It presents an oversimplified caricature of modern frequentist methods, and is

This is a great book. However, I really hate the "Golem of Prague" introduction. It presents an oversimplified caricature of modern frequentist methods, and is therefore rather misleading about the benefits of Bayesian modeling. Moreover, most practicing statisticians don't really view these points of view as incompatible. Compare to the treatment in Gelman et al.'s Bayesian Data Analysis. There are p-values all over the place. Most importantly, this critique fails on basic philosophical grounds. Suppose you give me a statistical problem, and I produce a Bayesian solution that, upon further examination with simulations, gives the wrong answer 90% of time on identical problems. If you think there's something wrong with that, then congratulations, you're a "frequentist," or at least believe there's some important insight about statistics that's not captured by doing everything in a rote Bayesian way. (And if you don't think there's something wrong with that, I'd love to hear why.) Also, this isn't a purely academic thought experiment. There are real examples of Bayesian estimators, for concrete and practical problems such as clustering, that give the wrong estimates for parameters with high probability (even as the sample size grows arbitrarily large).

kkoncevicius 5y ago
> If you think there's something wrong with that, then congratulations, you're a "frequentist". And more than that - if you use bootstrap, or do cross-validation, you are being a frequentist.
nightski 5y ago
Hmm. He was comparing the Bayesian models to golems as well, not just frequentist. It was an analogy to all statistical models. Second in the lectures he said that he uses frequentist techniques all the time and that it's often worth looking at it from each perspective. I interpreted it as his problem is not with the methods themselves, but with how they are commonly used in science. To me this made a lot of sense.
- spekcular 5y ago
  I think I'm misremembering. I read through some of the introductory material in the second edition of his book and found it less critical than I recalled. But in some places, it definitely comes across as hostile (e.g. footnote 107). Also, the sentence "Bayesian probability is a very general approach to probability, and it includes as a special case another important approach, the frequentist approach" is pretty funny. I know the exact technical result he's referring to, but it's clearly wrong to gloss it like that. He does mention consistency once, page 221, but (unconvincingly) handwaves away concerns about it. (Large N regimes exist that aren't N=infinity...)
  - nightski 5y ago
    Honestly I think it is a little hostile. Not towards frequentist directly, but towards the mis-use of frequentist methods in science. He works in ecology and I think he comes across a bunch of crap all the time. He talks at length about the statistical crisis in science and I can't really blame him. But I could see how someone might take this as an attack on the methods themselves.
    - kmonad 5y ago
      I agree. The golem is presented as an analogue to any statistical inference: powerful but ultimately dumb, in the sense that it won't think for you. That's in my opinion the major theme of the book---you have to think and not rely on algorithms/tools/machines...or golems to do that for you.
rwilson4 5y ago
Gill's book, Bayesian Methods, is even more dismissive, and even hostile towards Frequentist methods. Whereas I've never seen a frequentist book dismissive of Bayes methods. (Counterexamples welcome!) It boils down to whether you give precedence to the likelihood principle or the strong repeated sampling principle (Bayes prefers the likelihood principle and Frequentist prefers repeated sampling). See Cox and Hinkley's Theoretical Statistics for a full discussion, but basically the likelihood principle states that all conclusions should be based exclusively on the likelihood function; in layman's terms, on the data themselves. This specifically omits what a frequentist would call important contextual metadata, like whether the sample size is random, why the sample size is what it is, etc. The strong repeated sampling principle states that the goodness of a statistical procedure should be evaluated based on performance under hypothetical repetitions. Bayesians often dismiss this as: "what are these hypothetical repetitions? Why should I care?" Well, it depends. If you're predicting the results of an election, it's a special 1 time event. It isn't obvious what a repetition would mean. If you're analyzing an A/B test it's easy to imagine running another test, some other team running the same test, etc. Frequentist statistics values consistency here, more so than Bayesian methods do. That's not to come out in support of one vs the other. You need to understand the strengths and drawbacks of each and decide situationally which to use. (Disclaimer: I consider myself a Frequentist but sometimes use Bayesian methods.)
- spekcular 5y ago
  "Whereas I've never seen a frequentist book dismissive of Bayes methods. (Counterexamples welcome!)" Indeed! There's a lot of Bayesian propaganda floating around these days. While I enjoy it, I would also love to see some frequentist propaganda (ideally with substantive educational content...).
  - kkoncevicius 5y ago
    A book by Deborah Mayo "Statistical Inference as Severe Testing" might fit.
    - spekcular 5y ago
kgwgk 5y ago
Suppose you give me a particle physics problem, and I produce a quantum mechanics solution that, upon further examination, is wrong. If you think there's something wrong with that, then congratulations, you're a "quantum negationist," or at least believe there's some important insight about physics that's not captured by doing everything in a rote quantum way. (The important insight being that GIGO.)
- spekcular 5y ago
  The issue isn't that Bayesian methods used incorrectly can have bad frequentist properties. It's that, according to many flavors of Bayesianism, having bad frequentist properties isn't a valid line of critique. You may not believe in the particular stances I'm calling out, but if so, we don't disagree.
  - kgwgk 5y ago
    Maybe we don't disagree. You wrote: > a Bayesian solution that, upon further examination with simulations, gives the wrong answer 90% of time on identical problems If "with simulations" means either "with simulations using a probability distribution different from the prior used in the Bayesian analysis" or "with simulations using a model different from the one used in the Bayesian analysis" are we expected to conclude that there is something wrong with the Bayesian way?
    - spekcular 5y ago
      I mean "with simulations using a probability distribution [for the true parameter] different from the prior used in the Bayesian analysis." (The issue of model error is a separate question.) Yes, in this case would should conclude there is something wrong with the Bayesian way. If you hand me a statistical method to e.g. estimate some parameter that frequently returns answers that are far from the truth, that is a problem. One cannot assume the prior exactly describes reality (or there would be no point in doing inference, because the prior already gives you the truth).
      - 5y ago
agucova 5y ago
I think the classes opt for starting with a simple mental model students can adopt, which is gradually replaced with a more robust and nuanced mental model. In this case he wasn't talking just about frequentist methods tho, it's also talking about doing statistics without first doing science (and formulating a causal model). I would be wary of jumping to conclusions from that introduction alone if you haven't seen the rest of the course or the book.
funklute 5y ago
> There are real examples of Bayesian estimators, for concrete and practical problems such as clustering, that give the wrong estimates for parameters with high probability (even as the sample size grows arbitrarily large). Could you give some specific examples, and/or references? This is new to me, and I would like to read deeper into it.
- spekcular 5y ago
  I was thinking of this paper in particular: https://www.jmlr.org/papers/volume15/miller14a/miller14a.pdf https://www.jmlr.org/papers/volume15/miller14a/miller14a.pdf Some other references: https://www.jstor.org/stable/2241268 https://www.jstor.org/stable/2241268 https://arxiv.org/abs/1308.6306 https://arxiv.org/abs/1308.6306 https://projecteuclid.org/journals/annals-of-mathematical-statistics/volume-36/issue-2/On-the-Asymptotic-Behavior-of-Bayes-Estimates-in-the-Discrete/10.1214/aoms/1177700155.full https://projecteuclid.org/journals/annals-of-mathematical-st...
  - mturmon 5y ago
    Thanks for the detail! I took a look at the first paper, the result was new to me. In the vogue days of reversible jump MCMC I played with mixture estimation of the number of components under a basic prior (an approach which gives decent results in Figs 1 and 3), but I never used a Dirichlet process prior for this problem. This paper points out that even this simple approach is problematic because it’s only consistent if the true distribution is such a mixture, and in my case it definitely was not. Anyway, one takeaway, esp. from sec 1.2.1, is that the Dirichlet process prior is not suitable for estimating #components in most cases; it favors small clusters. And indeed, the concept of estimating #components is tricky to begin with, as noted above. Just because you can compute the posterior, doesn’t mean it’s saying what you think it is about the underlying true distribution!
medstrom 5y ago
Uh, dude. If you read the book, you'd see the Golem of Prague isn't a parable about frequentist models specifically, it's about all models, period. He calls his Bayesian models golems all the time.
throwawayarnty 5y ago
Agreed. There are situations where frequentist guarantees is what you want rather than optimal estimates of parameters. Frequentists consider the model parameters fixed and allow the data to vary. Bayesian consider the data fixed and allow parameters to vary. For example many software engineering, drug testing, and large scale pipelines want frequentist guarantees, because your system will have varying input data and you want theoretical bounds on what inferences you can make.

kkoncevicius 5y ago

> If you think there's something wrong with that, then congratulations, you're a "frequentist". And more than that - if you use bootstrap, or do cross-validation, you are being a frequentist.

nightski 5y ago

Hmm. He was comparing the Bayesian models to golems as well, not just frequentist. It was an analogy to all statistical models. Second in the lectures he said that he uses frequentist techniques all the time and that it's often worth looking at it from each perspective. I interpreted it as his problem is not with the methods themselves, but with how they are commonly used in science. To me this made a lot of sense.

spekcular 5y ago
I think I'm misremembering. I read through some of the introductory material in the second edition of his book and found it less critical than I recalled. But in some places, it definitely comes across as hostile (e.g. footnote 107). Also, the sentence "Bayesian probability is a very general approach to probability, and it includes as a special case another important approach, the frequentist approach" is pretty funny. I know the exact technical result he's referring to, but it's clearly wrong to gloss it like that. He does mention consistency once, page 221, but (unconvincingly) handwaves away concerns about it. (Large N regimes exist that aren't N=infinity...)
- nightski 5y ago
  Honestly I think it is a little hostile. Not towards frequentist directly, but towards the mis-use of frequentist methods in science. He works in ecology and I think he comes across a bunch of crap all the time. He talks at length about the statistical crisis in science and I can't really blame him. But I could see how someone might take this as an attack on the methods themselves.
  - kmonad 5y ago
    I agree. The golem is presented as an analogue to any statistical inference: powerful but ultimately dumb, in the sense that it won't think for you. That's in my opinion the major theme of the book---you have to think and not rely on algorithms/tools/machines...or golems to do that for you.

rwilson4 5y ago

Gill's book, Bayesian Methods, is even more dismissive, and even hostile towards Frequentist methods. Whereas I've never seen a frequentist book dismissive of Bayes methods. (Counterexamples welcome!) It boils down to whether you give precedence to the likelihood principle or the strong repeated sampling principle (Bayes prefers the likelihood principle and Frequentist prefers repeated sampling). See Cox and Hinkley's Theoretical Statistics for a full discussion, but basically the likelihood principle states that all conclusions should be based exclusively on the likelihood function; in layman's terms, on the data themselves. This specifically omits what a frequentist would call important contextual metadata, like whether the sample size is random, why the sample size is what it is, etc. The strong repeated sampling principle states that the goodness of a statistical procedure should be evaluated based on performance under hypothetical repetitions. Bayesians often dismiss this as: "what are these hypothetical repetitions? Why should I care?" Well, it depends. If you're predicting the results of an election, it's a special 1 time event. It isn't obvious what a repetition would mean. If you're analyzing an A/B test it's easy to imagine running another test, some other team running the same test, etc. Frequentist statistics values consistency here, more so than Bayesian methods do. That's not to come out in support of one vs the other. You need to understand the strengths and drawbacks of each and decide situationally which to use. (Disclaimer: I consider myself a Frequentist but sometimes use Bayesian methods.)

spekcular 5y ago
"Whereas I've never seen a frequentist book dismissive of Bayes methods. (Counterexamples welcome!)" Indeed! There's a lot of Bayesian propaganda floating around these days. While I enjoy it, I would also love to see some frequentist propaganda (ideally with substantive educational content...).
- kkoncevicius 5y ago
  A book by Deborah Mayo "Statistical Inference as Severe Testing" might fit.
  - spekcular 5y ago

kgwgk 5y ago

Suppose you give me a particle physics problem, and I produce a quantum mechanics solution that, upon further examination, is wrong. If you think there's something wrong with that, then congratulations, you're a "quantum negationist," or at least believe there's some important insight about physics that's not captured by doing everything in a rote quantum way. (The important insight being that GIGO.)

spekcular 5y ago
The issue isn't that Bayesian methods used incorrectly can have bad frequentist properties. It's that, according to many flavors of Bayesianism, having bad frequentist properties isn't a valid line of critique. You may not believe in the particular stances I'm calling out, but if so, we don't disagree.
- kgwgk 5y ago
  Maybe we don't disagree. You wrote: > a Bayesian solution that, upon further examination with simulations, gives the wrong answer 90% of time on identical problems If "with simulations" means either "with simulations using a probability distribution different from the prior used in the Bayesian analysis" or "with simulations using a model different from the one used in the Bayesian analysis" are we expected to conclude that there is something wrong with the Bayesian way?
  - spekcular 5y ago
    I mean "with simulations using a probability distribution [for the true parameter] different from the prior used in the Bayesian analysis." (The issue of model error is a separate question.) Yes, in this case would should conclude there is something wrong with the Bayesian way. If you hand me a statistical method to e.g. estimate some parameter that frequently returns answers that are far from the truth, that is a problem. One cannot assume the prior exactly describes reality (or there would be no point in doing inference, because the prior already gives you the truth).
    - 5y ago

agucova 5y ago

I think the classes opt for starting with a simple mental model students can adopt, which is gradually replaced with a more robust and nuanced mental model. In this case he wasn't talking just about frequentist methods tho, it's also talking about doing statistics without first doing science (and formulating a causal model). I would be wary of jumping to conclusions from that introduction alone if you haven't seen the rest of the course or the book.

funklute 5y ago

> There are real examples of Bayesian estimators, for concrete and practical problems such as clustering, that give the wrong estimates for parameters with high probability (even as the sample size grows arbitrarily large). Could you give some specific examples, and/or references? This is new to me, and I would like to read deeper into it.

spekcular 5y ago
I was thinking of this paper in particular: https://www.jmlr.org/papers/volume15/miller14a/miller14a.pdf https://www.jmlr.org/papers/volume15/miller14a/miller14a.pdf Some other references: https://www.jstor.org/stable/2241268 https://www.jstor.org/stable/2241268 https://arxiv.org/abs/1308.6306 https://arxiv.org/abs/1308.6306 https://projecteuclid.org/journals/annals-of-mathematical-statistics/volume-36/issue-2/On-the-Asymptotic-Behavior-of-Bayes-Estimates-in-the-Discrete/10.1214/aoms/1177700155.full https://projecteuclid.org/journals/annals-of-mathematical-st...
- mturmon 5y ago
  Thanks for the detail! I took a look at the first paper, the result was new to me. In the vogue days of reversible jump MCMC I played with mixture estimation of the number of components under a basic prior (an approach which gives decent results in Figs 1 and 3), but I never used a Dirichlet process prior for this problem. This paper points out that even this simple approach is problematic because it’s only consistent if the true distribution is such a mixture, and in my case it definitely was not. Anyway, one takeaway, esp. from sec 1.2.1, is that the Dirichlet process prior is not suitable for estimating #components in most cases; it favors small clusters. And indeed, the concept of estimating #components is tricky to begin with, as noted above. Just because you can compute the posterior, doesn’t mean it’s saying what you think it is about the underlying true distribution!

medstrom 5y ago

Uh, dude. If you read the book, you'd see the Golem of Prague isn't a parable about frequentist models specifically, it's about all models, period. He calls his Bayesian models golems all the time.

throwawayarnty 5y ago

Agreed. There are situations where frequentist guarantees is what you want rather than optimal estimates of parameters. Frequentists consider the model parameters fixed and allow the data to vary. Bayesian consider the data fixed and allow parameters to vary. For example many software engineering, drug testing, and large scale pipelines want frequentist guarantees, because your system will have varying input data and you want theoretical bounds on what inferences you can make.