8 ms·

Bayesian Statistics: The three cultures

309 points by luu 2y ago

thegginthesky 2y ago
I miss the college days where professors would argue endlessly on Bayesian vs Frequentist. The article is very well succinct and even explains why even my Bayesian professors had different approaches to research and analysis. I never knew about the third camp, Pragmatic Bayes, but definitely is in line with a professor's research that was very through on probability fit and the many iteration to get the prior and joint PDF just right. Andrew Gelman has a very cool talk "Andrew Gelman - Bayes, statistics, and reproducibility (Rutgers, Foundations of Probability)", which I highly recommend for many Data Scientists
- spootze 2y ago
  Regarding the frequentist vs bayesian debates, my slightly provocative take on these three cultures is - subjective Bayes is the strawman that frequentist academics like to attack - objective Bayes is a naive self-image that many Bayesian academics tend to possess - pragmatic Bayes is the approach taken by practitioners that actually apply statistics to something (or in Gelman’s terms, do science)
  - refulgentis 2y ago
    I see, so academics are frequentists (attackers) or objective Bayes (naive), and the people Doing Science are pragmatic (correct). The article gave me the same vibe, nice, short set of labels for me to apply as a heuristic. I never really understood this particular war, I'm a simpleton, A in Stats 101, that's it. I guess I need to bone up on Wikipedia to understand what's going on here more.
    - Yossarrian22 2y ago
      Academics can be pragmatic, I've know ones who've sued both Bayesian statistics and MLE
    - sgt101 2y ago
      Bayes lets you use your priors, which can be very helpful. I got all riled up when I saw you wrote "correct", I can't really explain why... but I just feel that we need to keep an open mind. These approaches to data are choices at the end of the day... Was Einstein a Bayesian? (spoiler: no)
      - refulgentis 2y ago
        You're absolutely right, trying to walk a delicate tightrope that doesn't end up with me giving my unfiltered "you're wrong so lets end conversation" response. Me 6 months ago would have written: "this comment is unhelpful and boring, but honestly, that's slightly unfair to you, as it just made me realize how little help the article is, and it set the tone. is this even a real argument with sides?" For people who want to improve on this aspect of themselves, like I did for years: - show, don't tell (ex. here, I made the oddities more explicit, enough that people could reply to me spelling out what I shouldn't.) - Don't assert anything that wasn't said directly, ex. don't remark on the commenter, or subjective qualities you assess in the comment.
      - 0cf8612b2e1e 2y ago
        Using your priors is another way of saying you know something about the problem. It is exceedingly difficult to objectively analyze a dataset without interjecting any bias. There are too many decision points where something needs to be done to massage the data into shape. Priors is just an explicit encoding of some of that knowledge.
        ants_everywhere 2y ago
        > Priors is just an explicit encoding of some of that knowledge. A classic example is analyzing data on mind reading or ghost detection. Your experiment shows you that your ghost detector has detected a haunting with p < .001. What is the probability the house is haunted?
        lupusreal 2y ago
        With a prior like that, why would you even bother pretending to do the research?
    - thegginthesky 2y ago
      Frequentist and Bayesian are correct if both have scientific rigor in their research and methodology. Both can be wrong if the research is whack or sloppy.
      - slashdave 2y ago
        I've used both in some papers and report two results (why not?). The golden rule in my mind is to fully describe your process and assumptions, then let the reader decide.
    - runarberg 2y ago
      I understand the war between bayesians and frequentists. Frequentist methods have been misused for over a century now to justify all sorts of pseudoscience and hoaxes (as well as created a fair share of honest mistakes), so it is understandable that people would come forward and claim there must be a better way. What I don’t understand is the war between naive bayes and pragmatic bayes. If it is real, it seems like the extension of philosophers vs. engineers. Scientists should see value in both. Naive Bayes is important to the philosophy of science, without which there would be a lot of junk science which would go unscrutinized for far to long, and engineers should be able to see the value of philosophers saving them works by debunking wrong science before they start to implement theories which simply will not work in practice.
  - DebtDeflation 2y ago
    A few things I wish I knew when took Statistics courses at university some 25 or so years ago: - Statistical significance testing and hypothesis testing are two completely different approaches with different philosophies behind them developed by different groups of people that kinda do the same thing but not quite and textbooks tend to completely blur this distinction out. - The above approaches were developed in the early 1900s in the context of farms and breweries where 3 things were true - 1) data was extremely limited, often there were only 5 or 6 data points available, 2) there were no electronic computers, so computation was limited to pen and paper and slide rules, and 3) the cost in terms of time and money of running experiments (e.g., planting a crop differently and waiting for harvest) were enormous. - The majority of classical statistics was focused on two simple questions - 1) what can I reliably say about a population based on a sample taken from it and 2) what can I reliably about the differences between two populations based on the samples taken from each? That's it. An enormous mathematical apparatus was built around answering those two questions in the context of the limitations in point #2.
    - ivansavz 2y ago
      That was a nice summary. The data-poor and computation-poor context of old school statistics definitely biased the methods towards the "recipe" approach scientists are supposed to follow mechanically, where each recipe is some predefined sequence of steps, justified based on an analytical approximations to a sampling distribution (given lots of assumptions). In modern computation-rich days, we can get away from the recipes by using resampling methods (e.g. permutation tests and bootstrap), so we don't need the analytical approximation formulas anymore. I think there is still room for small sample methods though... it's not like biological and social sciences are dealing with very large samples.
    - lottin 2y ago
      My understanding is that frequentist statistics was developed in response to the Bayesian methodology which was prevalent in the 1800s and which was starting to be perceived as having important flaws. The idea that the invention of Bayesian statistics made frequentist statistics obsolete doesn't quite agree with the historical facts.
  - skissane 2y ago
    > - subjective Bayes is the strawman that frequentist academics like to attack I don’t get what all the hate for subjective Bayesianism is. It seems the most philosophically defensible approach, in that all it assumes is our own subjective judgements of likelihood, the idea that we can quantify them (however in exactly), and the idea (avoid Dutch books) that we want to be consistent (most people do). Whereas, objective Bayes is basically subjective Bayes from the viewpoint of an idealised perfectly rational agent - and “perfectly rational” seems philosophically a lot more expensive than anything subjective Bayes relies on.
- RandomThoughts3 2y ago
  I’m always puzzled by this because while I come from a country where the frequentist approach generally dominates, the fight with Bayesian basically doesn’t exist. That’s just a bunch of mathematical theories and tools. Just use what’s useful. I’m still convinced that Americans tend to dislike the frequentist view because it requires a stronger background in mathematics.
  - parpfish 2y ago
    I don’t think mathematical ability has much to do with it. I think it’s useful to break down the anti-Bayesians into statisticians and non-statistician scientists. The former are mathematically savvy enough to understand bayes but object on philosophical grounds; the later don’t care about the philosophy so much as they feel like an attack on frequentism is an attack on their previous research and they take it personally
    - mturmon 2y ago
      This is a reasonable heuristic. I studied in a program that (for both philosophical and practical reasons) questioned whether the Bayesian formalism should be applied as widely as it is. (Which for many people is, basically everywhere.) There are some cases, that do arise in practice, where you can’t impose a prior, and/or where the “Dutch book” arguments to justify Bayesian decisions don’t apply.
  - thegginthesky 2y ago
    It's because practicioners of one says that the other camp is wrong and question each other's methodologies. And in academia, questioning one's methodology is akin to saying one is dumb. To understand both camps I summarize like this. Frequentist statistics has very sound theory but is misapplied by using many heuristics, rule of thumbs and prepared tables. It's very easy to use any method and hack the p-value away to get statistically significant results. Bayesian statistics has an interesting premise and inference methods, but until recently with the advancements of computing power, it was near impossible to do simulations to validate the complex distributions used, the goodness of fit and so on. And even in the current year, some bayesian statisticians don't question the priors and iterate on their research. I recommend using methods both whenever it's convenient and fits the problem at hand.
- bunderbunder 2y ago
  Link to talk: https://youtu.be/xgUBdi2wcDI https://youtu.be/xgUBdi2wcDI
  - mturmon 2y ago
    Thank you. In fact, the whole talk series (https://foundationsofprobabilityseminar.com/ https://foundationsofprobabilityseminar.com/) and channel (https://www.youtube.com/@foundationsofprobabilitypa2408/videos https://www.youtube.com/@foundationsofprobabilitypa2408/vide...) seem interesting.
- 3abiton 2y ago
  Funny enough I also heard recently about Fiducial Statistics as a 3rd camp, an intriguing podcast episode 581 of super data science, with the EiC of Harvard Business Review.
brcmthrowaway 2y ago
Where does Deep Learning come in?
- thegginthesky 2y ago
  Most models are derived of Machine Learning principles that are a mix of classic probability theory, Frequentist and Bayesian statistics and lots of Computer Science fundamentals. But there have been advancements in Bayesian Inference and Bayesian Deep Learning, you should check the work of frameworks like Pyro (built on top of PyTorch) Edit: corrected my sentence, but see 0xdde reply for better info.
  - 0xdde 2y ago
    I could be wrong, but my sense is that ML has leaned Bayesian for a very long time. For example, even Bishop's widely used book from 2006 [1] is Bayesian. Not sure how Bayesian his new deep learning book is. [1] https://www.microsoft.com/en-us/research/publication/pattern-recognition-machine-learning/ https://www.microsoft.com/en-us/research/publication/pattern...
    - thegginthesky 2y ago
      I stand corrected! It was my impression that many methods used in ML such as Support Vector Machines, Decision Trees, Random Forests, Boosting, Bagging and so on have very deep roots in Frequentist Methods, although current CS implementations lean heavily on optimizations such as Gradient Descent. Giving a cursory look into Bishop's book I see that I am wrong, as there's deep root in Bayesian Inference as well. On another note, I find it very interesting that there's not a bigger emphasis on using the correct distributions in ML models, as the methods are much more concerned in optimizing objective functions.
- tfehring 2y ago
  An implicit shared belief of all of the practitioners the author mentions is that they attempt to construct models that correspond to some underlying "data generating process". Machine learning practitioners may use similar models or even the same models as Bayesian statisticians, but they tend to evaluate their models primarily or entirely based on their predictive performance, not on intuitions about why the data is taking on the values that it is. See Breiman's classic "Two Cultures" paper that this post's title is referencing: https://projecteuclid.org/journals/statistical-science/volume-16/issue-3/Statistical-Modeling--The-Two-Cultures-with-comments-and-a/10.1214/ss/1009213726.full https://projecteuclid.org/journals/statistical-science/volum...
tonymet 2y ago
A priori distributions are a form of stereotyping. How do people reconcile that?
- lupire 2y ago
  A Bayesian analysis lets you see how the posterior varies as a function of the prior, instead of forcing you to pick a prior before you start. The tighter the range of this function, the more confidence you have in the result. You can never know anything if you absolutely refuse to have a prior, because that gives division by 0 in the posterior.
- klysm 2y ago
  What? Maybe in a very specific context where you are modeling joint distributions of people and traits, but that’s barely a critique of the method itself.
  - tonymet 2y ago
    it's not a critique of the method
mjhay 2y ago
The great thing about Bayesian statistics is that it's subjective. You don't have to be in the subjectivist school. You can choose your own interpretation based on your (subjective) judgment. I think this is a strength of Bayesianism. Any statistical work is infused with the subjective judgement of individual humans. I think it is more objective to not shy away from this immutable fact.
- klysm 2y ago
  The appropriateness of each approach is very much a function of what is being modeled and the corresponding consequences for error.
  - mjhay 2y ago
    Of course. The best approach for a particular problem depends on your best judgment. I guess that means I'm in the pragmatist school in this article's nomenclature (I'm a big fan of Gelman and all the other stats folks there), but what one thinks is pragmatic is also subjective.
tfehring 2y ago
The author is claiming that Bayesians vary along two axes: (1) whether they generally try to inform their priors with their knowledge or beliefs about the world, and (2) whether they iterate on the functional form of the model based on its goodness-of-fit and the reasonableness and utility of its outputs. He then labels 3 of the 4 resulting combinations as follows: ┌───────────────┬───────────┬──────────────┐ │ │ iteration │ no iteration │ ├───────────────┼───────────┼──────────────┤ │ informative │ pragmatic │ subjective │ │ uninformative │ - │ objective │ └───────────────┴───────────┴──────────────┘ My main disagreement with this model is the empty bottom-left box - in fact, I think that's where most self-labeled Bayesians in industry fall: - Iterating on the functional form of the model (and therefore the assumed underlying data generating process) is generally considered obviously good and necessary, in my experience. - Priors are usually uninformative or weakly informative, partly because data is often big enough to overwhelm the prior. The need for iteration feels so obvious to me that the entire "no iteration" column feels like a straw man. But the author, who knows far more academic statisticians than I do, explicitly says that he had the same belief and "was shocked to learn that statisticians didn’t think this way."
- klysm 2y ago
  The no iteration thing is very real and I don’t think it’s even for particularly bad reasons. We iterate on models to make them better, by some definition of better. It’s no secret that scientific work is subject to rather perverse incentives around thresholds of significance and positive results. Publish or perish. Perverse incentives lead to perverse statistics. The iteration itself is sometimes viewed directly as a problem. The “garden of forking paths”, where the analysis depends on the data, is viewed as a direct cause for some of the statistical and epistemological crises in science today. Iteration itself isn’t inherently bad. It’s just that the objective function usually isn’t what we want from a scientific perspective. To those actually doing scientific work, I suspect iterating on their models feels like they’re doing something unfaithful. Furthermore, I believe a lot of these issues are strongly related to the flawed epistemological framework which many scientific fields seem to have converged: p<0.05 means it’s true, otherwise it’s false. edit: Perhaps another way to characterize this discomfort is by the number of degrees of freedom that the analyst controls. In a Bayesian context where we are picking priors either by belief or previous data, the analyst has a _lot_ of control over how the results come out the other end. I think this is why fields have trended towards a set of ‘standard’ tests instead of building good statistical models. These take most of the knobs out of the hands of the analyst, and generally are more conservative.
derbOac 2y ago
I never liked the clubs you were expected to put yourself in, what "side" you were on, or the idea that problems in science that we see today could somehow be reduced to the inferential philosophy you adopt. In a lot of ways I see myself as information-theoretic in orientation, so maybe objective Bayesian, although it's really neither frequentist nor Bayesian. This three cultures idea is a bit of slight of hand in my opinion, as the "pragmatic" culture isn't really exclusive of subjective or objective Bayesianism and in that sense says nothing about how you should approach prior specification or interpretation or anything. Maybe Gelman would say a better term is "flexibility" or something but then that leaves the question of when you go objective and when you go subjective and why. Seems better to formalize that than leave it as a bit of smoke and mirrors. I'm not saying some flexibility about prior interpretation and specification isn't a good idea, just that I'm not sure that approaching theoretical basics with the answer "we'll just ignore the issues and pretend we're doing something different" is quite the right answer. Playing a bit of devil's advocate too, the "pragmatic" culture reveals a bit about why Bayesianism is looked at with a bit of skepticism and doubt. "Choosing a prior" followed by "seeing how well everything fits" and then "repeating" looks a lot like model tweaking or p-hacking. I know that's not the intent, and it's impossible to do modeling without tweaking, but if you approach things that way, the prior just looks like one more degree of freedom to nudge things around and fish with. I've published and edited papers on Bayesian inference, and my feeling is that the problems with it have never been in the theory, which is solid. It's in how people use and abuse it in practice.
bayesian_trout 2y ago
If you want to get an informed opinion on modern Frequentist methods check out the book "In All Likelihood" by Yudi Pawitawn. In an early chapter it outlines, rather eloquently, the distinctions between the Frequentist and Bayesian paradigms and in particular the power of well-designed Frequentist or likelihood-based models. With few exceptions, an analyst should get the same answer using a Bayesian vs. Frequentist model if the Bayesian is actually using uninformative priors. In the worlds I work in, 99% of the time I see researchers using Bayesian methods they are also claiming to use uninformative priors, which makes me wonder if they are just using Bayesian methods to sound cool and skip through peer review. One potential problem with Bayesian statistics lies in the fact that for complicated models (100s or even 1000s of parameters) it can be extremely difficult to know if the priors are truly uninformative in the context of a particular dataset. One has to wait for models to run, and when systematically changing priors this can take an extraordinary amount of time, even when using high powered computing resources. Additionally, in the Bayesian setting it becomes easy to accidentally "glue" a model together with a prior or set of priors that would simply bomb out and give a non-positive definite hessian in the Frequentist world (read: a diagnostic telling you that your model is likely bogus and/or too complex for a given dataset). One might scoff at models of this complexity, but that is the reality in many applied settings, for example spatio-temporal models facing the "big n" problem or for stuff like integrated fisheries assessment models used to assess status and provide information on stock sustainability. So my primary beef with Bayesian statistics (and I say this as someone who teaches graduate level courses on the Bayesian inference) is that it can very easily be misused by non-statisticians and beginners, particularly given the extremely flexible software programs that currently are available to non-statisticians like biologists etc. In general though, both paradigms are subjective and Gelman's argument that it is turtles (i.e., subjectivity) all the way down is spot on and really resonates with me.
- usgroup 2y ago
  +1 for “in all likelihood” but it should be stated that the book explains a third approach which doesn’t lean on either subjective or objective probability.
  - 2y ago
deleted 2y ago
[deleted]
davidgerard 2y ago
> Subjective Bayes > I’m not sure if anyone ever followed this philosophy strictly, nor do I know if anyone would register their affiliation as subjective Bayesian these days. lol the lesswrong/rationalist "Bayesians" do this all the time. * I have priors * YOU have biases * HE is a toxoplasmotic culture warrior
prmph 2y ago
So my theory is that probability is an ill-defined, unfalsifiable concept. And yet, it _seems_ to model aspects of the world pretty well, empirically. However, might it be leading us astray? Consider the statement p(X) = 0.5 (probability of event X is 0.5). What does this actually mean? It it a proposition? If so, is it falsifiable? And how? If it is not a proposition, what does it actually mean? If someone with more knowledge can chime in here, I'd be grateful. I've got much more to say on this, but only after I hear from those with a rigorous grounding the theory.
- kgwgk 2y ago
  > If it is not a proposition, what does it actually mean? It's a measure of plausibility - enabling plausible reasoning. https://www.lesswrong.com/posts/KN3BYDkWei9ADXnBy/e-t-jaynes-probability-theory-the-logic-of-science-i https://www.lesswrong.com/posts/KN3BYDkWei9ADXnBy/e-t-jaynes... https://en.wikipedia.org/wiki/Cox%27s_theorem https://en.wikipedia.org/wiki/Cox%27s_theorem
- ants_everywhere 2y ago
  So here's a sort of hard-nosed answer: probability is just as well-defined as any other mathematics. > Consider the statement p(X) = 0.5 (probability of event X is 0.5). What does this actually mean? It means X is a random variable from some sample space to a measurable space and P is a probability function. > If so, is it falsifiable? And how? Yes, by calculating P(X) in the given sample space. For example, if X is the event "you get 100 heads in a row when flipping a fair coin" then it is false that P(X) = 0.5. It's a bit like asking whether 2^2 = 4 is falsifiable. There are definitely meaningful questions to ask about whether you've modeled the problem correctly, just as it's meaningful to ask what "2" and "4" mean. But those are separate questions from whether the statements of probability are falsifiable. If you can show that the probability axioms hold for your problem, then you can use probability theory on it. There's a Wikipedia article on interpretations of probability here: https://en.wikipedia.org/wiki/Probability_interpretations https://en.wikipedia.org/wiki/Probability_interpretations. But it is pretty short and doesn't seem quite so complete.
  - prmph 2y ago
usgroup 2y ago
Bare in mind that Breiman's polemic was about generative vs discriminative methods. I.e. that we should not start an analysis by thinking about how the data generation can be modelled, but instead we should start with prediction. From that vein came boosted trees, bagging, random forests, xgboost and so on: non generative black box methods. Still today most of the classical machine learning toolbox is not generative.
- gwd 2y ago
  Nit: "Bear in mind". "Bare" means "to make bare" (i.e., to uncover); "bear" means "to carry": "As you evaluate this discussion, carry it in your mind that..."
nurettin 2y ago
Bayesian hacking: Find the iteration that gives the most significance to your research.

309 points by luu 2y ago

I miss the college days where professors would argue endlessly on Bayesian vs Frequentist. The article is very well succinct and even explains why even my Bayesian professors had different approaches to research and analysis. I never knew about the third camp, Pragmatic Bayes, but definitely is in line with a professor's research that was very through on probability fit and the many iteration to get the prior and joint PDF just right. Andrew Gelman has a very cool talk "Andrew Gelman - Bayes, statistics, and reproducibility (Rutgers, Foundations of Probability)", which I highly recommend for many Data Scientists

spootze 2y ago
Regarding the frequentist vs bayesian debates, my slightly provocative take on these three cultures is - subjective Bayes is the strawman that frequentist academics like to attack - objective Bayes is a naive self-image that many Bayesian academics tend to possess - pragmatic Bayes is the approach taken by practitioners that actually apply statistics to something (or in Gelman’s terms, do science)
- refulgentis 2y ago
  I see, so academics are frequentists (attackers) or objective Bayes (naive), and the people Doing Science are pragmatic (correct). The article gave me the same vibe, nice, short set of labels for me to apply as a heuristic. I never really understood this particular war, I'm a simpleton, A in Stats 101, that's it. I guess I need to bone up on Wikipedia to understand what's going on here more.
  - Yossarrian22 2y ago
    Academics can be pragmatic, I've know ones who've sued both Bayesian statistics and MLE
  - sgt101 2y ago
    Bayes lets you use your priors, which can be very helpful. I got all riled up when I saw you wrote "correct", I can't really explain why... but I just feel that we need to keep an open mind. These approaches to data are choices at the end of the day... Was Einstein a Bayesian? (spoiler: no)
    - refulgentis 2y ago
      You're absolutely right, trying to walk a delicate tightrope that doesn't end up with me giving my unfiltered "you're wrong so lets end conversation" response. Me 6 months ago would have written: "this comment is unhelpful and boring, but honestly, that's slightly unfair to you, as it just made me realize how little help the article is, and it set the tone. is this even a real argument with sides?" For people who want to improve on this aspect of themselves, like I did for years: - show, don't tell (ex. here, I made the oddities more explicit, enough that people could reply to me spelling out what I shouldn't.) - Don't assert anything that wasn't said directly, ex. don't remark on the commenter, or subjective qualities you assess in the comment.
    - 0cf8612b2e1e 2y ago
      Using your priors is another way of saying you know something about the problem. It is exceedingly difficult to objectively analyze a dataset without interjecting any bias. There are too many decision points where something needs to be done to massage the data into shape. Priors is just an explicit encoding of some of that knowledge.
      - ants_everywhere 2y ago
        > Priors is just an explicit encoding of some of that knowledge. A classic example is analyzing data on mind reading or ghost detection. Your experiment shows you that your ghost detector has detected a haunting with p < .001. What is the probability the house is haunted?
        lupusreal 2y ago
        With a prior like that, why would you even bother pretending to do the research?
  - thegginthesky 2y ago
    Frequentist and Bayesian are correct if both have scientific rigor in their research and methodology. Both can be wrong if the research is whack or sloppy.
    - slashdave 2y ago
      I've used both in some papers and report two results (why not?). The golden rule in my mind is to fully describe your process and assumptions, then let the reader decide.
  - runarberg 2y ago
    I understand the war between bayesians and frequentists. Frequentist methods have been misused for over a century now to justify all sorts of pseudoscience and hoaxes (as well as created a fair share of honest mistakes), so it is understandable that people would come forward and claim there must be a better way. What I don’t understand is the war between naive bayes and pragmatic bayes. If it is real, it seems like the extension of philosophers vs. engineers. Scientists should see value in both. Naive Bayes is important to the philosophy of science, without which there would be a lot of junk science which would go unscrutinized for far to long, and engineers should be able to see the value of philosophers saving them works by debunking wrong science before they start to implement theories which simply will not work in practice.
- DebtDeflation 2y ago
  A few things I wish I knew when took Statistics courses at university some 25 or so years ago: - Statistical significance testing and hypothesis testing are two completely different approaches with different philosophies behind them developed by different groups of people that kinda do the same thing but not quite and textbooks tend to completely blur this distinction out. - The above approaches were developed in the early 1900s in the context of farms and breweries where 3 things were true - 1) data was extremely limited, often there were only 5 or 6 data points available, 2) there were no electronic computers, so computation was limited to pen and paper and slide rules, and 3) the cost in terms of time and money of running experiments (e.g., planting a crop differently and waiting for harvest) were enormous. - The majority of classical statistics was focused on two simple questions - 1) what can I reliably say about a population based on a sample taken from it and 2) what can I reliably about the differences between two populations based on the samples taken from each? That's it. An enormous mathematical apparatus was built around answering those two questions in the context of the limitations in point #2.
  - ivansavz 2y ago
    That was a nice summary. The data-poor and computation-poor context of old school statistics definitely biased the methods towards the "recipe" approach scientists are supposed to follow mechanically, where each recipe is some predefined sequence of steps, justified based on an analytical approximations to a sampling distribution (given lots of assumptions). In modern computation-rich days, we can get away from the recipes by using resampling methods (e.g. permutation tests and bootstrap), so we don't need the analytical approximation formulas anymore. I think there is still room for small sample methods though... it's not like biological and social sciences are dealing with very large samples.
  - lottin 2y ago
    My understanding is that frequentist statistics was developed in response to the Bayesian methodology which was prevalent in the 1800s and which was starting to be perceived as having important flaws. The idea that the invention of Bayesian statistics made frequentist statistics obsolete doesn't quite agree with the historical facts.
- skissane 2y ago
  > - subjective Bayes is the strawman that frequentist academics like to attack I don’t get what all the hate for subjective Bayesianism is. It seems the most philosophically defensible approach, in that all it assumes is our own subjective judgements of likelihood, the idea that we can quantify them (however in exactly), and the idea (avoid Dutch books) that we want to be consistent (most people do). Whereas, objective Bayes is basically subjective Bayes from the viewpoint of an idealised perfectly rational agent - and “perfectly rational” seems philosophically a lot more expensive than anything subjective Bayes relies on.
RandomThoughts3 2y ago
I’m always puzzled by this because while I come from a country where the frequentist approach generally dominates, the fight with Bayesian basically doesn’t exist. That’s just a bunch of mathematical theories and tools. Just use what’s useful. I’m still convinced that Americans tend to dislike the frequentist view because it requires a stronger background in mathematics.
- parpfish 2y ago
  I don’t think mathematical ability has much to do with it. I think it’s useful to break down the anti-Bayesians into statisticians and non-statistician scientists. The former are mathematically savvy enough to understand bayes but object on philosophical grounds; the later don’t care about the philosophy so much as they feel like an attack on frequentism is an attack on their previous research and they take it personally
  - mturmon 2y ago
    This is a reasonable heuristic. I studied in a program that (for both philosophical and practical reasons) questioned whether the Bayesian formalism should be applied as widely as it is. (Which for many people is, basically everywhere.) There are some cases, that do arise in practice, where you can’t impose a prior, and/or where the “Dutch book” arguments to justify Bayesian decisions don’t apply.
- thegginthesky 2y ago
  It's because practicioners of one says that the other camp is wrong and question each other's methodologies. And in academia, questioning one's methodology is akin to saying one is dumb. To understand both camps I summarize like this. Frequentist statistics has very sound theory but is misapplied by using many heuristics, rule of thumbs and prepared tables. It's very easy to use any method and hack the p-value away to get statistically significant results. Bayesian statistics has an interesting premise and inference methods, but until recently with the advancements of computing power, it was near impossible to do simulations to validate the complex distributions used, the goodness of fit and so on. And even in the current year, some bayesian statisticians don't question the priors and iterate on their research. I recommend using methods both whenever it's convenient and fits the problem at hand.
bunderbunder 2y ago
Link to talk: https://youtu.be/xgUBdi2wcDI https://youtu.be/xgUBdi2wcDI
- mturmon 2y ago
  Thank you. In fact, the whole talk series (https://foundationsofprobabilityseminar.com/ https://foundationsofprobabilityseminar.com/) and channel (https://www.youtube.com/@foundationsofprobabilitypa2408/videos https://www.youtube.com/@foundationsofprobabilitypa2408/vide...) seem interesting.
3abiton 2y ago
Funny enough I also heard recently about Fiducial Statistics as a 3rd camp, an intriguing podcast episode 581 of super data science, with the EiC of Harvard Business Review.

brcmthrowaway 2y ago

Where does Deep Learning come in?

thegginthesky 2y ago
Most models are derived of Machine Learning principles that are a mix of classic probability theory, Frequentist and Bayesian statistics and lots of Computer Science fundamentals. But there have been advancements in Bayesian Inference and Bayesian Deep Learning, you should check the work of frameworks like Pyro (built on top of PyTorch) Edit: corrected my sentence, but see 0xdde reply for better info.
- 0xdde 2y ago
  I could be wrong, but my sense is that ML has leaned Bayesian for a very long time. For example, even Bishop's widely used book from 2006 [1] is Bayesian. Not sure how Bayesian his new deep learning book is. [1] https://www.microsoft.com/en-us/research/publication/pattern-recognition-machine-learning/ https://www.microsoft.com/en-us/research/publication/pattern...
  - thegginthesky 2y ago
    I stand corrected! It was my impression that many methods used in ML such as Support Vector Machines, Decision Trees, Random Forests, Boosting, Bagging and so on have very deep roots in Frequentist Methods, although current CS implementations lean heavily on optimizations such as Gradient Descent. Giving a cursory look into Bishop's book I see that I am wrong, as there's deep root in Bayesian Inference as well. On another note, I find it very interesting that there's not a bigger emphasis on using the correct distributions in ML models, as the methods are much more concerned in optimizing objective functions.
tfehring 2y ago
An implicit shared belief of all of the practitioners the author mentions is that they attempt to construct models that correspond to some underlying "data generating process". Machine learning practitioners may use similar models or even the same models as Bayesian statisticians, but they tend to evaluate their models primarily or entirely based on their predictive performance, not on intuitions about why the data is taking on the values that it is. See Breiman's classic "Two Cultures" paper that this post's title is referencing: https://projecteuclid.org/journals/statistical-science/volume-16/issue-3/Statistical-Modeling--The-Two-Cultures-with-comments-and-a/10.1214/ss/1009213726.full https://projecteuclid.org/journals/statistical-science/volum...

tonymet 2y ago

A priori distributions are a form of stereotyping. How do people reconcile that?

lupire 2y ago
A Bayesian analysis lets you see how the posterior varies as a function of the prior, instead of forcing you to pick a prior before you start. The tighter the range of this function, the more confidence you have in the result. You can never know anything if you absolutely refuse to have a prior, because that gives division by 0 in the posterior.
klysm 2y ago
What? Maybe in a very specific context where you are modeling joint distributions of people and traits, but that’s barely a critique of the method itself.
- tonymet 2y ago
  it's not a critique of the method

mjhay 2y ago

The great thing about Bayesian statistics is that it's subjective. You don't have to be in the subjectivist school. You can choose your own interpretation based on your (subjective) judgment. I think this is a strength of Bayesianism. Any statistical work is infused with the subjective judgement of individual humans. I think it is more objective to not shy away from this immutable fact.

klysm 2y ago
The appropriateness of each approach is very much a function of what is being modeled and the corresponding consequences for error.
- mjhay 2y ago
  Of course. The best approach for a particular problem depends on your best judgment. I guess that means I'm in the pragmatist school in this article's nomenclature (I'm a big fan of Gelman and all the other stats folks there), but what one thinks is pragmatic is also subjective.

tfehring 2y ago

The author is claiming that Bayesians vary along two axes: (1) whether they generally try to inform their priors with their knowledge or beliefs about the world, and (2) whether they iterate on the functional form of the model based on its goodness-of-fit and the reasonableness and utility of its outputs. He then labels 3 of the 4 resulting combinations as follows: ┌───────────────┬───────────┬──────────────┐ │ │ iteration │ no iteration │ ├───────────────┼───────────┼──────────────┤ │ informative │ pragmatic │ subjective │ │ uninformative │ - │ objective │ └───────────────┴───────────┴──────────────┘ My main disagreement with this model is the empty bottom-left box - in fact, I think that's where most self-labeled Bayesians in industry fall: - Iterating on the functional form of the model (and therefore the assumed underlying data generating process) is generally considered obviously good and necessary, in my experience. - Priors are usually uninformative or weakly informative, partly because data is often big enough to overwhelm the prior. The need for iteration feels so obvious to me that the entire "no iteration" column feels like a straw man. But the author, who knows far more academic statisticians than I do, explicitly says that he had the same belief and "was shocked to learn that statisticians didn’t think this way."

klysm 2y ago
The no iteration thing is very real and I don’t think it’s even for particularly bad reasons. We iterate on models to make them better, by some definition of better. It’s no secret that scientific work is subject to rather perverse incentives around thresholds of significance and positive results. Publish or perish. Perverse incentives lead to perverse statistics. The iteration itself is sometimes viewed directly as a problem. The “garden of forking paths”, where the analysis depends on the data, is viewed as a direct cause for some of the statistical and epistemological crises in science today. Iteration itself isn’t inherently bad. It’s just that the objective function usually isn’t what we want from a scientific perspective. To those actually doing scientific work, I suspect iterating on their models feels like they’re doing something unfaithful. Furthermore, I believe a lot of these issues are strongly related to the flawed epistemological framework which many scientific fields seem to have converged: p<0.05 means it’s true, otherwise it’s false. edit: Perhaps another way to characterize this discomfort is by the number of degrees of freedom that the analyst controls. In a Bayesian context where we are picking priors either by belief or previous data, the analyst has a _lot_ of control over how the results come out the other end. I think this is why fields have trended towards a set of ‘standard’ tests instead of building good statistical models. These take most of the knobs out of the hands of the analyst, and generally are more conservative.

derbOac 2y ago

I never liked the clubs you were expected to put yourself in, what "side" you were on, or the idea that problems in science that we see today could somehow be reduced to the inferential philosophy you adopt. In a lot of ways I see myself as information-theoretic in orientation, so maybe objective Bayesian, although it's really neither frequentist nor Bayesian. This three cultures idea is a bit of slight of hand in my opinion, as the "pragmatic" culture isn't really exclusive of subjective or objective Bayesianism and in that sense says nothing about how you should approach prior specification or interpretation or anything. Maybe Gelman would say a better term is "flexibility" or something but then that leaves the question of when you go objective and when you go subjective and why. Seems better to formalize that than leave it as a bit of smoke and mirrors. I'm not saying some flexibility about prior interpretation and specification isn't a good idea, just that I'm not sure that approaching theoretical basics with the answer "we'll just ignore the issues and pretend we're doing something different" is quite the right answer. Playing a bit of devil's advocate too, the "pragmatic" culture reveals a bit about why Bayesianism is looked at with a bit of skepticism and doubt. "Choosing a prior" followed by "seeing how well everything fits" and then "repeating" looks a lot like model tweaking or p-hacking. I know that's not the intent, and it's impossible to do modeling without tweaking, but if you approach things that way, the prior just looks like one more degree of freedom to nudge things around and fish with. I've published and edited papers on Bayesian inference, and my feeling is that the problems with it have never been in the theory, which is solid. It's in how people use and abuse it in practice.

bayesian_trout 2y ago

If you want to get an informed opinion on modern Frequentist methods check out the book "In All Likelihood" by Yudi Pawitawn. In an early chapter it outlines, rather eloquently, the distinctions between the Frequentist and Bayesian paradigms and in particular the power of well-designed Frequentist or likelihood-based models. With few exceptions, an analyst should get the same answer using a Bayesian vs. Frequentist model if the Bayesian is actually using uninformative priors. In the worlds I work in, 99% of the time I see researchers using Bayesian methods they are also claiming to use uninformative priors, which makes me wonder if they are just using Bayesian methods to sound cool and skip through peer review. One potential problem with Bayesian statistics lies in the fact that for complicated models (100s or even 1000s of parameters) it can be extremely difficult to know if the priors are truly uninformative in the context of a particular dataset. One has to wait for models to run, and when systematically changing priors this can take an extraordinary amount of time, even when using high powered computing resources. Additionally, in the Bayesian setting it becomes easy to accidentally "glue" a model together with a prior or set of priors that would simply bomb out and give a non-positive definite hessian in the Frequentist world (read: a diagnostic telling you that your model is likely bogus and/or too complex for a given dataset). One might scoff at models of this complexity, but that is the reality in many applied settings, for example spatio-temporal models facing the "big n" problem or for stuff like integrated fisheries assessment models used to assess status and provide information on stock sustainability. So my primary beef with Bayesian statistics (and I say this as someone who teaches graduate level courses on the Bayesian inference) is that it can very easily be misused by non-statisticians and beginners, particularly given the extremely flexible software programs that currently are available to non-statisticians like biologists etc. In general though, both paradigms are subjective and Gelman's argument that it is turtles (i.e., subjectivity) all the way down is spot on and really resonates with me.

usgroup 2y ago
+1 for “in all likelihood” but it should be stated that the book explains a third approach which doesn’t lean on either subjective or objective probability.
- 2y ago

deleted 2y ago

[deleted]

davidgerard 2y ago

> Subjective Bayes > I’m not sure if anyone ever followed this philosophy strictly, nor do I know if anyone would register their affiliation as subjective Bayesian these days. lol the lesswrong/rationalist "Bayesians" do this all the time. * I have priors * YOU have biases * HE is a toxoplasmotic culture warrior

prmph 2y ago

So my theory is that probability is an ill-defined, unfalsifiable concept. And yet, it _seems_ to model aspects of the world pretty well, empirically. However, might it be leading us astray? Consider the statement p(X) = 0.5 (probability of event X is 0.5). What does this actually mean? It it a proposition? If so, is it falsifiable? And how? If it is not a proposition, what does it actually mean? If someone with more knowledge can chime in here, I'd be grateful. I've got much more to say on this, but only after I hear from those with a rigorous grounding the theory.

kgwgk 2y ago
> If it is not a proposition, what does it actually mean? It's a measure of plausibility - enabling plausible reasoning. https://www.lesswrong.com/posts/KN3BYDkWei9ADXnBy/e-t-jaynes-probability-theory-the-logic-of-science-i https://www.lesswrong.com/posts/KN3BYDkWei9ADXnBy/e-t-jaynes... https://en.wikipedia.org/wiki/Cox%27s_theorem https://en.wikipedia.org/wiki/Cox%27s_theorem
ants_everywhere 2y ago
So here's a sort of hard-nosed answer: probability is just as well-defined as any other mathematics. > Consider the statement p(X) = 0.5 (probability of event X is 0.5). What does this actually mean? It means X is a random variable from some sample space to a measurable space and P is a probability function. > If so, is it falsifiable? And how? Yes, by calculating P(X) in the given sample space. For example, if X is the event "you get 100 heads in a row when flipping a fair coin" then it is false that P(X) = 0.5. It's a bit like asking whether 2^2 = 4 is falsifiable. There are definitely meaningful questions to ask about whether you've modeled the problem correctly, just as it's meaningful to ask what "2" and "4" mean. But those are separate questions from whether the statements of probability are falsifiable. If you can show that the probability axioms hold for your problem, then you can use probability theory on it. There's a Wikipedia article on interpretations of probability here: https://en.wikipedia.org/wiki/Probability_interpretations https://en.wikipedia.org/wiki/Probability_interpretations. But it is pretty short and doesn't seem quite so complete.
- prmph 2y ago

usgroup 2y ago

Bare in mind that Breiman's polemic was about generative vs discriminative methods. I.e. that we should not start an analysis by thinking about how the data generation can be modelled, but instead we should start with prediction. From that vein came boosted trees, bagging, random forests, xgboost and so on: non generative black box methods. Still today most of the classical machine learning toolbox is not generative.

gwd 2y ago
Nit: "Bear in mind". "Bare" means "to make bare" (i.e., to uncover); "bear" means "to carry": "As you evaluate this discussion, carry it in your mind that..."

nurettin 2y ago

Bayesian hacking: Find the iteration that gives the most significance to your research.