4 ms·

The Bayesian Cringe (2021)

65 points by EndXA 3y ago

hackburg 3y ago
[dead]
olesya1979 3y ago
[dead]
parpfish 3y ago
I noticed a shift in my attitude in strong priors when I switched from academia to industry and have only recently realized why. When doing an analysis in an academic setting, the goal is to get a paper past reviewers to be published. And the reviewers were adversaries that were trying to disprove your work (at best these were helpful critique; at worst they were bad-faith nit-pickers that were looking for any excuse to reject). If you did a Bayesian analysis in this setting, the mean reviewers would just just point to the priors and say "you can't justify that choice, REJECT". But in industry, there are no reviewers serving as adversarial gatekeepers. You may present analyses to a skeptical audience, but if they disagreed with your model priors you would work with them to come up with a mutually agreeable model because you're all on the same team.
- silent_cal 3y ago
  It's funny because there's really no difference between arbitrarily setting priors and arbitrarily selecting a p-value. Both are basically just a line you draw beforehand to see if it gets crossed.
- BugsJustFindMe 3y ago
  > But in industry, there are no reviewers serving as adversarial gatekeepers. You may present analyses to a skeptical audience, but if they disagreed with your model priors you would work with them to come up with a mutually agreeable model because you're all on the same team. This experience may not be representative. The web is absolutely filled with anecdotes of gatekeeping and obstructionism within engineering orgs. The phrase "internal politics" comes immediately to mind.
  - whatshisface 3y ago
    Internal politics have nothing to do with the content, so I think their point stands. There's no reason not to do a Bayesian analysis, if your proposal fails it will certainly be for another reason.
- vundercind 3y ago
  Most folks’ experience of primary and secondary education, and maybe also undergrad, is similar, with the instructor as the adversary. How much more collaborative and forgiving the “scary” “real world” is, was a real surprise after so many years of school.
  - whatshisface 3y ago
    It can easily go too far though, especial when somebody doesn't want to accept criticism. Then you're not a team player if you check the arithmetic.
- williamdclt 3y ago
  It’s definitely a learned soft skill to direct interactions to be collaborative rather than adversarial. It’s too easy to fall into an adversarial discussion because of differing opinions (eg about code architecture) when really you’re on the same team. I try to keep in mind (and convey) the image of “you and me side to side against the problem on the whiteboard” rather than “you and me against each other”
- 3abiton 3y ago
  Really interesting insight, yo add, in academia you are trying to prove yourself to the reviewers, wheras in business settings, you are offering services. Different goals.
sk11001 3y ago
I still have no clue what Gelman is saying about anything ever, and this post is no exeception. He seems like a great guy in interviews and presentations but anything he writes or talks about is highly non-specific.
- uoaei 3y ago
  I find that it's rather specific and niche, and maybe that's why you aren't getting it.
bruturis 3y ago
Prior are now not subjective but useful, the OP is about the problem of choosing the best priors. The best options are informative priors (1) and regularizers (2). So, for example, choosing as prior a Laplace distribution for the unknown parameters is equivalent to the LASSO that is a well known way of obtaining sparse models with few coefficients. In (2) there is an example in which a prior suggest a useful regularization method for regression. In (3) the author discusses prior modeling. (1) https://en.wikipedia.org/wiki/Prior_probability#Informative_priors https://en.wikipedia.org/wiki/Prior_probability#Informative_... (2) https://skeptric.com/prior-regularise/index.html https://skeptric.com/prior-regularise/index.html (3) https://betanalpha.github.io/assets/case_studies/prior_modeling.html https://betanalpha.github.io/assets/case_studies/prior_model...
kqr 3y ago
There's also the fact that a prior is really hard to explain to someone else. By definition, it's the unexplainable starting point! Yet when I lay out fairly tight Bayesian reasoning, there's always that one person sucking life out of the entire conversation with "Wait can you go back to that first number? How did you arrive at that?" and it's an unanswerable question because any attempt would have to start from another, more fundamental prior! Sometimes this person is reasonable and I can go, "Ah, we can try a different starting point. What's your prior?" but often enough the person gets stuck on the idea of subjective probability and everything derails. When it comes to important decisions, I've started hiding the prior with smoke and mirrors to redirect attention away from it.
- silent_cal 3y ago
  And yet nobody asks "how did you come up with that .05 significance level?"
- perrygeo 3y ago
  I hoped we were past the point where people describe priors as "subjective". Sigh. Bayes formula can be applied with complete objectivity. Inference given the same starting knowledge plus the same evidence will always yield the same conclusion. It need not be subjective, but it IS relative to one's knowledge. And why is that a bad thing? Shouldn't more knowledge == better inference? We have a systematic way to build knowledge with reasonable objectivity (the scientific method) - should we not use it? Or do these people literally believe that less knowledge somehow improves their decision making?
  - mashygpig 3y ago
    It seems that there’s still unavoidable subjectivity in making the choice of prior distribution? I get how it’s objective for a fixed choice, but my understanding is that you need to first make that choice in order to be objective. Is it actually that making your choice of prior is obvious (or there is some objectively optimal way to pick a prior), which rules out any subjectivity in the choice of prior?
oldgradstudent 3y ago
What's the difference between a prior and a bias? How does one distinguish between the two?
- silent_cal 3y ago
  You can set your priors equally. 100% / number of possible outcomes.
- layer8 3y ago
  A prior is whatever you start with. There’s literally no requirements. Bayes tells you how to update your priors, whatever they are, in the face of new data. Nothing more, nothing less. In principle it doesn’t matter what priors you start with (how biased they are), in the sense that given enough data, your likelihoods will converge to what is really the case.
  - biomcgary 3y ago
    If you have the right model.
mayd 3y ago
I thought that a good Bayesian model was supposed to be demonstrably robust under a range of prior distributions?
- JHonaker 3y ago
  Yes, but a "range of prior distributions" doesn't mean every possible prior distribution. Sometimes, the information in the prior distribution is required to get you to a place where your computational system can efficiently explore a meaningful subspace instead of providing nonsense. If meaningfully different priors lead to meaningfully different posteriors, you're probably missing something that would either eliminate one of those priors from contention or marry the differing behavior in some unifying explanation/model. Either way is a win in my book; both provide a new direction for research!
theGnuMe 3y ago
Le Cam had interesting comments on Bayesian statistics. https://errorstatistics.com/2013/11/18/lucien-le-cam-the-bayesians-hold-the-magic/ https://errorstatistics.com/2013/11/18/lucien-le-cam-the-bay...

65 points by EndXA 3y ago

hackburg 3y ago

[dead]

olesya1979 3y ago

[dead]

parpfish 3y ago

I noticed a shift in my attitude in strong priors when I switched from academia to industry and have only recently realized why. When doing an analysis in an academic setting, the goal is to get a paper past reviewers to be published. And the reviewers were adversaries that were trying to disprove your work (at best these were helpful critique; at worst they were bad-faith nit-pickers that were looking for any excuse to reject). If you did a Bayesian analysis in this setting, the mean reviewers would just just point to the priors and say "you can't justify that choice, REJECT". But in industry, there are no reviewers serving as adversarial gatekeepers. You may present analyses to a skeptical audience, but if they disagreed with your model priors you would work with them to come up with a mutually agreeable model because you're all on the same team.

silent_cal 3y ago
It's funny because there's really no difference between arbitrarily setting priors and arbitrarily selecting a p-value. Both are basically just a line you draw beforehand to see if it gets crossed.
BugsJustFindMe 3y ago
> But in industry, there are no reviewers serving as adversarial gatekeepers. You may present analyses to a skeptical audience, but if they disagreed with your model priors you would work with them to come up with a mutually agreeable model because you're all on the same team. This experience may not be representative. The web is absolutely filled with anecdotes of gatekeeping and obstructionism within engineering orgs. The phrase "internal politics" comes immediately to mind.
- whatshisface 3y ago
  Internal politics have nothing to do with the content, so I think their point stands. There's no reason not to do a Bayesian analysis, if your proposal fails it will certainly be for another reason.
vundercind 3y ago
Most folks’ experience of primary and secondary education, and maybe also undergrad, is similar, with the instructor as the adversary. How much more collaborative and forgiving the “scary” “real world” is, was a real surprise after so many years of school.
- whatshisface 3y ago
  It can easily go too far though, especial when somebody doesn't want to accept criticism. Then you're not a team player if you check the arithmetic.
williamdclt 3y ago
It’s definitely a learned soft skill to direct interactions to be collaborative rather than adversarial. It’s too easy to fall into an adversarial discussion because of differing opinions (eg about code architecture) when really you’re on the same team. I try to keep in mind (and convey) the image of “you and me side to side against the problem on the whiteboard” rather than “you and me against each other”
3abiton 3y ago
Really interesting insight, yo add, in academia you are trying to prove yourself to the reviewers, wheras in business settings, you are offering services. Different goals.

sk11001 3y ago

I still have no clue what Gelman is saying about anything ever, and this post is no exeception. He seems like a great guy in interviews and presentations but anything he writes or talks about is highly non-specific.

uoaei 3y ago
I find that it's rather specific and niche, and maybe that's why you aren't getting it.

bruturis 3y ago

Prior are now not subjective but useful, the OP is about the problem of choosing the best priors. The best options are informative priors (1) and regularizers (2). So, for example, choosing as prior a Laplace distribution for the unknown parameters is equivalent to the LASSO that is a well known way of obtaining sparse models with few coefficients. In (2) there is an example in which a prior suggest a useful regularization method for regression. In (3) the author discusses prior modeling. (1) https://en.wikipedia.org/wiki/Prior_probability#Informative_priors https://en.wikipedia.org/wiki/Prior_probability#Informative_... (2) https://skeptric.com/prior-regularise/index.html https://skeptric.com/prior-regularise/index.html (3) https://betanalpha.github.io/assets/case_studies/prior_modeling.html https://betanalpha.github.io/assets/case_studies/prior_model...

kqr 3y ago

There's also the fact that a prior is really hard to explain to someone else. By definition, it's the unexplainable starting point! Yet when I lay out fairly tight Bayesian reasoning, there's always that one person sucking life out of the entire conversation with "Wait can you go back to that first number? How did you arrive at that?" and it's an unanswerable question because any attempt would have to start from another, more fundamental prior! Sometimes this person is reasonable and I can go, "Ah, we can try a different starting point. What's your prior?" but often enough the person gets stuck on the idea of subjective probability and everything derails. When it comes to important decisions, I've started hiding the prior with smoke and mirrors to redirect attention away from it.

silent_cal 3y ago
And yet nobody asks "how did you come up with that .05 significance level?"
perrygeo 3y ago
I hoped we were past the point where people describe priors as "subjective". Sigh. Bayes formula can be applied with complete objectivity. Inference given the same starting knowledge plus the same evidence will always yield the same conclusion. It need not be subjective, but it IS relative to one's knowledge. And why is that a bad thing? Shouldn't more knowledge == better inference? We have a systematic way to build knowledge with reasonable objectivity (the scientific method) - should we not use it? Or do these people literally believe that less knowledge somehow improves their decision making?
- mashygpig 3y ago
  It seems that there’s still unavoidable subjectivity in making the choice of prior distribution? I get how it’s objective for a fixed choice, but my understanding is that you need to first make that choice in order to be objective. Is it actually that making your choice of prior is obvious (or there is some objectively optimal way to pick a prior), which rules out any subjectivity in the choice of prior?

oldgradstudent 3y ago

What's the difference between a prior and a bias? How does one distinguish between the two?

silent_cal 3y ago
You can set your priors equally. 100% / number of possible outcomes.
layer8 3y ago
A prior is whatever you start with. There’s literally no requirements. Bayes tells you how to update your priors, whatever they are, in the face of new data. Nothing more, nothing less. In principle it doesn’t matter what priors you start with (how biased they are), in the sense that given enough data, your likelihoods will converge to what is really the case.
- biomcgary 3y ago
  If you have the right model.

mayd 3y ago

I thought that a good Bayesian model was supposed to be demonstrably robust under a range of prior distributions?

JHonaker 3y ago
Yes, but a "range of prior distributions" doesn't mean every possible prior distribution. Sometimes, the information in the prior distribution is required to get you to a place where your computational system can efficiently explore a meaningful subspace instead of providing nonsense. If meaningfully different priors lead to meaningfully different posteriors, you're probably missing something that would either eliminate one of those priors from contention or marry the differing behavior in some unifying explanation/model. Either way is a win in my book; both provide a new direction for research!

theGnuMe 3y ago

Le Cam had interesting comments on Bayesian statistics. https://errorstatistics.com/2013/11/18/lucien-le-cam-the-bayesians-hold-the-magic/ https://errorstatistics.com/2013/11/18/lucien-le-cam-the-bay...