5 ms·

I've been reading polemics and tutorials for at least 12 years now arguing for Bayesian methods over Frequentist methods. They all seem persuasive, and everyone

I've been reading polemics and tutorials for at least 12 years now arguing for Bayesian methods over Frequentist methods. They all seem persuasive, and everyone seems to be convinced that Bayesianism is the future, and Frequentism was a mistake, and the change to a glorious future where Bayesian methods are the standard way to do stats is just round the corner. It hasn't happened. Meanwhile I've never read a real argument for Frequentism and I don't know where I'd find one, short of going back to Fisher who is not well known for clear writing. What is going on? Is the future of Bayesian statistics just more and more decades of books and articles and code notebooks and great presentations showing how great Bayesianism is? Is it just institutional inertia preventing Bayesian stats from becoming standard, or does Frequentism have a secret strength that keeps it hegemonic?

agucova 5y ago
I guess it really depends on your discipline, but Bayesian methods have become more and more popular in a lot of academic communities, being published alongside papers using frequentist methods, so I wouldn't say it's hegemonic anymore.
lkozma 5y ago
For those coming from a CS background a possible (crude) intuition sometimes given is that frequentist :: Bayesian ~ worst-case analysis :: average-case analysis There are a good reasons why we don't usually do average-case analysis of algorithms, chief among them that we have no idea how inputs are distributed (another reason is computational difficulty). Worst-case bounds are pessimistic, but they hold.
- zozbot234 5y ago
  In CS you're often dealing with adversarial inputs, for which worst-case analysis is obviously the right approach. Not sure that there's anything comparable to that in most statistical settings.
civilized 5y ago
The biggest problem with Bayesian statistics in practice is the frequent reliance on relatively slow, unreliable methods such as MCMC. The Bayesian methodology community loves to advocate for packages like Stan, claiming that they make Bayesian stats easy. This is true... relative to Bayesian stats without Stan. But these packages are often much, much harder to get useful results from than the methods of classical statistics. You have to worry about all sorts of technical issues specific to these methodologies, because these issues don't have general technical solutions. Knowing when you even have a solution is often a huge pain with sampling techniques like MCMC. So you have to become an expert in this zoo of Bayesian technicalities above and beyond whatever actual problem you are trying to solve. And in the end, the results usually carry no more insight or reliability than you would have gotten from a simpler method. I recommend learning about Bayesianism at a philosophical level. Every scientist should know how to evaluate their results from a Bayesian perspective, at least qualitatively. But don't get too into Bayesian methodology beyond simple methods like conjugate prior updating... unless you are lucky enough to have a problem that is amenable to a reliable, practical Bayesian solution.
- deleted 5y ago
  [deleted]
- analog31 5y ago
  >>> Every scientist should know how to evaluate their results from a Bayesian perspective, at least qualitatively. I'm a scientist. How do I do this? I took one stats course, more than 30 years ago, and it was mostly proofs. Of course we learned Bayes' Theorem, and solved some problems using it. What I don't remember is being taught to choose an "ism" for doing math, so it could easily be something that I'm already doing and taking for granted.
  - civilized 5y ago
    At a minimum, recognize that not all p < 0.05 results have an equal claim on scientific truth. The study power and the prior probability of the hypothesis being true also influence the posterior probability [1]. In particular, underpowered studies of implausible hypotheses provide almost no scientific signal [2], and p < 0.05 is meaningless in such cases. [1] https://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124 https://journals.plos.org/plosmedicine/article?id=10.1371/jo... [2] http://www.stat.columbia.edu/~gelman/research/unpublished/power4r.pdf http://www.stat.columbia.edu/~gelman/research/unpublished/po...
13415 5y ago
I don't know much about statistical uses of Bayesianism but can say something opinionated about the underlying philosophy. From a philosophical point of view, Bayesianism is fairly weak and lacks argumentative support. The underlying idea of probabilism - that degrees of belief have to be represented by probability measures - is in my opinion wrong for many reasons. Basically the only well-developed arguments for this view are Dutch book arguments, which make a number of questionable assumptions. Besides, priors are also often not known. As far as I can see, subjective utilities can only be considered rational as long as they match objective probabilities, i.e., if the agent responds in epistemically truth-conducive ways (using successful learning methods) to evidence and does not have strongly misleading and skewed priors. I also reject the use of simple probability representations in decision theory, first because they do not adequately represent uncertainty, second because they make too strong rationality assumptions in the multiattribute case, and third because there are good reasons why evaluations of outcomes and states of affairs ought to be based on lexicographic value comparisons, not just on a simple expected utility principle. Generally speaking, Bayesians in this area tend to choose too simple epistemic representations and too simple value representations. The worst kind of Bayesians in philosophy are those who present Bayesian updating as if it was the only right way to respond to evidence. This is wrong on many levels, most notably by misunderstanding how theory discovery can and should work. In contrast, frequentism is way more cautious and does not make weird normative-psychological claims about how our beliefs ought to be structured. It represents an overall more skeptical approach, especially when hypothesis testing is combined with causal models. A propensity analysis of probability may also sometimes make sense, but this depends on analytical models and these are not always available. There are good uses of Bayesian statistics that do not hinge on subjective probabilities and any of the above philosophical views about them, and for which the priors are well motivated. But the philosophical underpinnings are weak, and whenever I read an application of Bayesian statistics I first wonder whether the authors haven't just used this method to do some trickery that might be problematic at a closer look. I'd be happy if everyone would just use classical hypothesis testing in a pre-registered study with a p value below 1%.
- kgwgk 5y ago
  > The underlying idea of probabilism - that degrees of belief have to be represented by probability measures - is in my opinion wrong for many reasons. Basically the only well-developed arguments for this view are Dutch book arguments, which make a number of questionable assumptions. Why don't you consider Cox's theorem - and related arguments - well-developed? https://en.wikipedia.org/wiki/Cox%27s_theorem https://en.wikipedia.org/wiki/Cox%27s_theorem
cossatot 5y ago
Institutional and community resistance shouldn’t be underestimated. Until about a decade ago, it could be really hard to get papers accepted in my scientific field (geosciences, particularly geology) that used Bayesian methods because senior faculty and journal editors deemed them subjective and less rigorous than frequentist ‘letter tests’. This didn’t really change until more powerful computers and mathy scripting languages allowed the younger scientists to generate undeniable better results, particularly for very poorly conditioned inverse and parameter estimation problems. The wave of pro-Bayesian propaganda has been promulgated in no small part by the younger scientists who have been fighting for acceptance for a long time, but the resistance fought through editorial and grant proposal rejection letters, not blog posts, more powerful but less noisy. It’s also still unlikely to find Bayes stats classes in most geoscience departments, so practitioners are largely self-taught outside of Michigan, Caltech etc. I imagine it’s been similar in many branches of science.
clircle 5y ago
> Meanwhile I've never read a real argument for Frequentism and I don't know where I'd find one, short of going back to Fisher who is not well known for clear writing. Check out the works of Deborah Mayo and her peers. She has several books and papers on the virtues of Frequentism that range from non technical to technical. Her blog is excellent as well. errorstatistics.com Also, check out modern formulations of frequentist models in terms of estimating equations, which add a certain level of coherence to frequentist models that approaches a Bayesian coherence.

agucova 5y ago

I guess it really depends on your discipline, but Bayesian methods have become more and more popular in a lot of academic communities, being published alongside papers using frequentist methods, so I wouldn't say it's hegemonic anymore.

lkozma 5y ago

For those coming from a CS background a possible (crude) intuition sometimes given is that frequentist :: Bayesian ~ worst-case analysis :: average-case analysis There are a good reasons why we don't usually do average-case analysis of algorithms, chief among them that we have no idea how inputs are distributed (another reason is computational difficulty). Worst-case bounds are pessimistic, but they hold.

zozbot234 5y ago
In CS you're often dealing with adversarial inputs, for which worst-case analysis is obviously the right approach. Not sure that there's anything comparable to that in most statistical settings.

civilized 5y ago

The biggest problem with Bayesian statistics in practice is the frequent reliance on relatively slow, unreliable methods such as MCMC. The Bayesian methodology community loves to advocate for packages like Stan, claiming that they make Bayesian stats easy. This is true... relative to Bayesian stats without Stan. But these packages are often much, much harder to get useful results from than the methods of classical statistics. You have to worry about all sorts of technical issues specific to these methodologies, because these issues don't have general technical solutions. Knowing when you even have a solution is often a huge pain with sampling techniques like MCMC. So you have to become an expert in this zoo of Bayesian technicalities above and beyond whatever actual problem you are trying to solve. And in the end, the results usually carry no more insight or reliability than you would have gotten from a simpler method. I recommend learning about Bayesianism at a philosophical level. Every scientist should know how to evaluate their results from a Bayesian perspective, at least qualitatively. But don't get too into Bayesian methodology beyond simple methods like conjugate prior updating... unless you are lucky enough to have a problem that is amenable to a reliable, practical Bayesian solution.

deleted 5y ago
[deleted]
analog31 5y ago
>>> Every scientist should know how to evaluate their results from a Bayesian perspective, at least qualitatively. I'm a scientist. How do I do this? I took one stats course, more than 30 years ago, and it was mostly proofs. Of course we learned Bayes' Theorem, and solved some problems using it. What I don't remember is being taught to choose an "ism" for doing math, so it could easily be something that I'm already doing and taking for granted.
- civilized 5y ago
  At a minimum, recognize that not all p < 0.05 results have an equal claim on scientific truth. The study power and the prior probability of the hypothesis being true also influence the posterior probability [1]. In particular, underpowered studies of implausible hypotheses provide almost no scientific signal [2], and p < 0.05 is meaningless in such cases. [1] https://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124 https://journals.plos.org/plosmedicine/article?id=10.1371/jo... [2] http://www.stat.columbia.edu/~gelman/research/unpublished/power4r.pdf http://www.stat.columbia.edu/~gelman/research/unpublished/po...

13415 5y ago

I don't know much about statistical uses of Bayesianism but can say something opinionated about the underlying philosophy. From a philosophical point of view, Bayesianism is fairly weak and lacks argumentative support. The underlying idea of probabilism - that degrees of belief have to be represented by probability measures - is in my opinion wrong for many reasons. Basically the only well-developed arguments for this view are Dutch book arguments, which make a number of questionable assumptions. Besides, priors are also often not known. As far as I can see, subjective utilities can only be considered rational as long as they match objective probabilities, i.e., if the agent responds in epistemically truth-conducive ways (using successful learning methods) to evidence and does not have strongly misleading and skewed priors. I also reject the use of simple probability representations in decision theory, first because they do not adequately represent uncertainty, second because they make too strong rationality assumptions in the multiattribute case, and third because there are good reasons why evaluations of outcomes and states of affairs ought to be based on lexicographic value comparisons, not just on a simple expected utility principle. Generally speaking, Bayesians in this area tend to choose too simple epistemic representations and too simple value representations. The worst kind of Bayesians in philosophy are those who present Bayesian updating as if it was the only right way to respond to evidence. This is wrong on many levels, most notably by misunderstanding how theory discovery can and should work. In contrast, frequentism is way more cautious and does not make weird normative-psychological claims about how our beliefs ought to be structured. It represents an overall more skeptical approach, especially when hypothesis testing is combined with causal models. A propensity analysis of probability may also sometimes make sense, but this depends on analytical models and these are not always available. There are good uses of Bayesian statistics that do not hinge on subjective probabilities and any of the above philosophical views about them, and for which the priors are well motivated. But the philosophical underpinnings are weak, and whenever I read an application of Bayesian statistics I first wonder whether the authors haven't just used this method to do some trickery that might be problematic at a closer look. I'd be happy if everyone would just use classical hypothesis testing in a pre-registered study with a p value below 1%.

kgwgk 5y ago
> The underlying idea of probabilism - that degrees of belief have to be represented by probability measures - is in my opinion wrong for many reasons. Basically the only well-developed arguments for this view are Dutch book arguments, which make a number of questionable assumptions. Why don't you consider Cox's theorem - and related arguments - well-developed? https://en.wikipedia.org/wiki/Cox%27s_theorem https://en.wikipedia.org/wiki/Cox%27s_theorem

cossatot 5y ago

Institutional and community resistance shouldn’t be underestimated. Until about a decade ago, it could be really hard to get papers accepted in my scientific field (geosciences, particularly geology) that used Bayesian methods because senior faculty and journal editors deemed them subjective and less rigorous than frequentist ‘letter tests’. This didn’t really change until more powerful computers and mathy scripting languages allowed the younger scientists to generate undeniable better results, particularly for very poorly conditioned inverse and parameter estimation problems. The wave of pro-Bayesian propaganda has been promulgated in no small part by the younger scientists who have been fighting for acceptance for a long time, but the resistance fought through editorial and grant proposal rejection letters, not blog posts, more powerful but less noisy. It’s also still unlikely to find Bayes stats classes in most geoscience departments, so practitioners are largely self-taught outside of Michigan, Caltech etc. I imagine it’s been similar in many branches of science.

clircle 5y ago

> Meanwhile I've never read a real argument for Frequentism and I don't know where I'd find one, short of going back to Fisher who is not well known for clear writing. Check out the works of Deborah Mayo and her peers. She has several books and papers on the virtues of Frequentism that range from non technical to technical. Her blog is excellent as well. errorstatistics.com Also, check out modern formulations of frequentist models in terms of estimating equations, which add a certain level of coherence to frequentist models that approaches a Bayesian coherence.