3 ms·

The biggest problem with Bayesian statistics in practice is the frequent reliance on relatively slow, unreliable methods such as MCMC. The Bayesian methodology

The biggest problem with Bayesian statistics in practice is the frequent reliance on relatively slow, unreliable methods such as MCMC. The Bayesian methodology community loves to advocate for packages like Stan, claiming that they make Bayesian stats easy. This is true... relative to Bayesian stats without Stan. But these packages are often much, much harder to get useful results from than the methods of classical statistics. You have to worry about all sorts of technical issues specific to these methodologies, because these issues don't have general technical solutions. Knowing when you even have a solution is often a huge pain with sampling techniques like MCMC. So you have to become an expert in this zoo of Bayesian technicalities above and beyond whatever actual problem you are trying to solve. And in the end, the results usually carry no more insight or reliability than you would have gotten from a simpler method. I recommend learning about Bayesianism at a philosophical level. Every scientist should know how to evaluate their results from a Bayesian perspective, at least qualitatively. But don't get too into Bayesian methodology beyond simple methods like conjugate prior updating... unless you are lucky enough to have a problem that is amenable to a reliable, practical Bayesian solution.

deleted 5y ago
[deleted]
analog31 5y ago
>>> Every scientist should know how to evaluate their results from a Bayesian perspective, at least qualitatively. I'm a scientist. How do I do this? I took one stats course, more than 30 years ago, and it was mostly proofs. Of course we learned Bayes' Theorem, and solved some problems using it. What I don't remember is being taught to choose an "ism" for doing math, so it could easily be something that I'm already doing and taking for granted.
- civilized 5y ago
  At a minimum, recognize that not all p < 0.05 results have an equal claim on scientific truth. The study power and the prior probability of the hypothesis being true also influence the posterior probability [1]. In particular, underpowered studies of implausible hypotheses provide almost no scientific signal [2], and p < 0.05 is meaningless in such cases. [1] https://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124 https://journals.plos.org/plosmedicine/article?id=10.1371/jo... [2] http://www.stat.columbia.edu/~gelman/research/unpublished/power4r.pdf http://www.stat.columbia.edu/~gelman/research/unpublished/po...
  - analog31 5y ago
    Check. Thankfully I come from the physics world, where we don't trust ourselves with any analysis method unless we have some independent way to make sure our results make sense. Most physicists are uneasy about "fancy" statistics. There's the famous quip: "If an experiment needs statistics, then one ought to have done a better experiment." While this is a bit extreme, it does represent a tendency to design experiments, when possible, that lend themselves to relatively basic data analysis. In fact, I haven't ever calculated a p value in my work. From what I've read about Bayesian statistics, I'm not sure it's going to get the life sciences out of the mess they're in. That's just my personal hunch. The trouble is, for justifiable reasons, life science research is hard, and is conducted with urgency due to the immediate need for solutions (such as vaccines, therapies for mental illness, or cynically, advertising revenue). A good rule of thumb seems to be: Multiply p by 10. In other words, p < 0.05 is a coin toss. This rule seems to replicate the entire replication crisis. ;-)

deleted 5y ago

[deleted]

analog31 5y ago

>>> Every scientist should know how to evaluate their results from a Bayesian perspective, at least qualitatively. I'm a scientist. How do I do this? I took one stats course, more than 30 years ago, and it was mostly proofs. Of course we learned Bayes' Theorem, and solved some problems using it. What I don't remember is being taught to choose an "ism" for doing math, so it could easily be something that I'm already doing and taking for granted.

civilized 5y ago
At a minimum, recognize that not all p < 0.05 results have an equal claim on scientific truth. The study power and the prior probability of the hypothesis being true also influence the posterior probability [1]. In particular, underpowered studies of implausible hypotheses provide almost no scientific signal [2], and p < 0.05 is meaningless in such cases. [1] https://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124 https://journals.plos.org/plosmedicine/article?id=10.1371/jo... [2] http://www.stat.columbia.edu/~gelman/research/unpublished/power4r.pdf http://www.stat.columbia.edu/~gelman/research/unpublished/po...
- analog31 5y ago
  Check. Thankfully I come from the physics world, where we don't trust ourselves with any analysis method unless we have some independent way to make sure our results make sense. Most physicists are uneasy about "fancy" statistics. There's the famous quip: "If an experiment needs statistics, then one ought to have done a better experiment." While this is a bit extreme, it does represent a tendency to design experiments, when possible, that lend themselves to relatively basic data analysis. In fact, I haven't ever calculated a p value in my work. From what I've read about Bayesian statistics, I'm not sure it's going to get the life sciences out of the mess they're in. That's just my personal hunch. The trouble is, for justifiable reasons, life science research is hard, and is conducted with urgency due to the immediate need for solutions (such as vaccines, therapies for mental illness, or cynically, advertising revenue). A good rule of thumb seems to be: Multiply p by 10. In other words, p < 0.05 is a coin toss. This rule seems to replicate the entire replication crisis. ;-)