5 ms·

There are other benefits to Bayesian data analysis besides being able to handle limited data. There are problems with the outputs of frequentist analysis around

There are other benefits to Bayesian data analysis besides being able to handle limited data. There are problems with the outputs of frequentist analysis around the quantification of uncertainty. For instance, from simulation studies we know that the aleatoric coverage probability for confidence intervals of a selected confidence level varies depending on the size of the difference in plausibility between the null and alternative hypotheses. And a given confidence interval says nothing about epistemic uncertainty for this particular experiment. This can make the outputs of frequentist analysis difficult for stakeholders to utilize, whereas Bayesian epistemic probabilities are generally more easily understand by stakeholders, and can directly feed quantitative decision analysis methods. A good introduction to some additional problems with frequentist methods vs Bayesian and likelihoodist methods is this: https://gandenberger.org/2014/08/26/intro-to-statistical-methods-3/ https://gandenberger.org/2014/08/26/intro-to-statistical-met... An interesting book on adapting frequentist methods to create confidence distributions that can better express uncertainty and can optionally incorporate prior information using likelihood functions is this: https://www.cambridge.org/core/books/confidence-likelihood-probability/143A34F11FB3D6F611F78E27C6D2CA5A#fndtn-information https://www.cambridge.org/core/books/confidence-likelihood-p...

wenc 3y ago
I’ve found that in many applications, the difference between a frequentist analysis and a Bayesian one is unlikely to make a difference in the decision making (even with UQ). I’m sure there are fields where such statistical rigor is called for (where the quality of the data is so high and accurate that the variation is in the analysis — often the case with machine data). For everything else there’s so much error. Being to quantify uncertainty is great — it’s a signal we need to collect more and better data. But so often we have to move ahead with uncertain data. Interestingly, in business, taking action (even if wrong) produces outcomes that are much better signals to learn from than having statistically rigorous analyses, so many times there’s a bias for action rather than obsession over analysis. But of course in some fields being wrong is costly (like clinical trials) so I can see UQ being more useful and prominent there.
- srean 3y ago
  In many of my projects I have had to incorporate the knowledge/intuition of domain experts. New product launch (by self or competitor), some unseen change in the operating environment. These are event history of the 'does not repeat but rhymes' variety. Although there may be no data collected from the time something similar happened before in history, experts can reason through the situation to guesstimate the direction and magnitude of the effect in qualitative terms. Bayesian formulations are very handy in such situations. >I’ve found that in many applications, the difference between a frequentist analysis and a Bayesian one is unlikely to make a difference in the decision making (even with UQ). In that case you may find the following interesting https://en.wikipedia.org/wiki/Lindley%27s_paradox https://en.wikipedia.org/wiki/Lindley%27s_paradox "Lindley's paradox is a counterintuitive situation in statistics in which the Bayesian and frequentist approaches to a hypothesis testing problem give different results for certain choices of the prior distribution." How likely is Lindley's Paradox likely to show up in practice ? well there is Bayes for that (tongue firmly in cheek).
  - wenc 3y ago
    Unless it’s a stacked analysis when results of one study depends on another, usually experts just eyeball the frequentist results and take a judgment call — that’s been my experience (not generalizing but I think in business people like doing things that are simple and easy to understand). I think it’s definitely possible that Bayesian and Frequentist approaches give different conclusions but in practice it doesn’t alter the final decision. Analyses guide decision making but in the end decisions are made on consensus, narrative and intuition. Statistics is only the handmaiden rather than the arbiter.
    - srean 3y ago
      > usually experts just eyeball the frequentist results and take a judgment call Indeed, but that does not make it right or rational. Bayesian method helps keep things rational. This is pertinent because human brains are terrible at conditional probabilities. One can always argue that data analysis is usually just window dressing and decision making is mostly political and social. Empirically you would be mostly right if you take that position. One cannot argue against that factual observation. The more interesting question is, if the decision makers aspire to be rational, which method should they use. I have used frequentist and Bayesian methods both. I made the choice on the basis of the question that needed answering. For example, when we needed to monitor (and alert on) a time varying probability of error (under time varying sample sizes) -- Bayesian method was a more natural fit than say confidence intervals or hypothesis tests. Bayesian methods directly address the question "What is the probability that error probability is below the threshold now, considering domain expert's opinion about how often it goes below the threshold and how the data has looked in the recent past?"

wenc 3y ago

I’ve found that in many applications, the difference between a frequentist analysis and a Bayesian one is unlikely to make a difference in the decision making (even with UQ). I’m sure there are fields where such statistical rigor is called for (where the quality of the data is so high and accurate that the variation is in the analysis — often the case with machine data). For everything else there’s so much error. Being to quantify uncertainty is great — it’s a signal we need to collect more and better data. But so often we have to move ahead with uncertain data. Interestingly, in business, taking action (even if wrong) produces outcomes that are much better signals to learn from than having statistically rigorous analyses, so many times there’s a bias for action rather than obsession over analysis. But of course in some fields being wrong is costly (like clinical trials) so I can see UQ being more useful and prominent there.

srean 3y ago
In many of my projects I have had to incorporate the knowledge/intuition of domain experts. New product launch (by self or competitor), some unseen change in the operating environment. These are event history of the 'does not repeat but rhymes' variety. Although there may be no data collected from the time something similar happened before in history, experts can reason through the situation to guesstimate the direction and magnitude of the effect in qualitative terms. Bayesian formulations are very handy in such situations. >I’ve found that in many applications, the difference between a frequentist analysis and a Bayesian one is unlikely to make a difference in the decision making (even with UQ). In that case you may find the following interesting https://en.wikipedia.org/wiki/Lindley%27s_paradox https://en.wikipedia.org/wiki/Lindley%27s_paradox "Lindley's paradox is a counterintuitive situation in statistics in which the Bayesian and frequentist approaches to a hypothesis testing problem give different results for certain choices of the prior distribution." How likely is Lindley's Paradox likely to show up in practice ? well there is Bayes for that (tongue firmly in cheek).
- wenc 3y ago
  Unless it’s a stacked analysis when results of one study depends on another, usually experts just eyeball the frequentist results and take a judgment call — that’s been my experience (not generalizing but I think in business people like doing things that are simple and easy to understand). I think it’s definitely possible that Bayesian and Frequentist approaches give different conclusions but in practice it doesn’t alter the final decision. Analyses guide decision making but in the end decisions are made on consensus, narrative and intuition. Statistics is only the handmaiden rather than the arbiter.
  - srean 3y ago
    > usually experts just eyeball the frequentist results and take a judgment call Indeed, but that does not make it right or rational. Bayesian method helps keep things rational. This is pertinent because human brains are terrible at conditional probabilities. One can always argue that data analysis is usually just window dressing and decision making is mostly political and social. Empirically you would be mostly right if you take that position. One cannot argue against that factual observation. The more interesting question is, if the decision makers aspire to be rational, which method should they use. I have used frequentist and Bayesian methods both. I made the choice on the basis of the question that needed answering. For example, when we needed to monitor (and alert on) a time varying probability of error (under time varying sample sizes) -- Bayesian method was a more natural fit than say confidence intervals or hypothesis tests. Bayesian methods directly address the question "What is the probability that error probability is below the threshold now, considering domain expert's opinion about how often it goes below the threshold and how the data has looked in the recent past?"