6 ms·

Bayes Rules – An Introduction to Applied Bayesian Modeling (2021)

147 points by RafelMri 4y ago

lemursage 4y ago
The scope seems pretty standard but I really like strongly structured form and exercises. Is there a pdf version available perchance?
platz 4y ago
The hardest part of bayes is 1) understanding what 'likelihood' actually is or represents 2) understanding what a 'partition function' actually is or represents I always forget what these mean since I don't use bayes in anger Also the above reveal there is more elaborate structure in bayes than just the idea of 'updating your prior with new information', which is simply a platitude among STEM folk
- DiogenesKynikos 4y ago
  > understanding what 'likelihood' actually is or represents The probability of observing the data you observed, assuming that the model parameters take on a certain value. > understanding what a 'partition function' actually is or represents The probability of observing the data you observed, this time averaging over all possible parameter values (weighted by the priors).
  - psi75 4y ago
    > The probability of observing the data you observed Yes, in discrete cases. In continuous cases, you have to work with a probability density. I think this is one of the hurdles people encounter when they're first exposed to Bayesian stats. The probabilities, technically speaking, are zero. The important insight in Bayesian work is that it's often not the probabilities themselves that matter but the ratios thereof, since from those alone you can compute posteriors.
    - DiogenesKynikos 4y ago
      Indeed, but it's a bit tedious to always say, "probabilities (in the discrete case) or probability densities (in the continuous case)." In general, you have sums in the discrete case and integrals in the continuous case, but most formulas are otherwise the same.
      - psi75 4y ago
        That's quite true. Also, one could argue that continuous probabilities in practice are discrete probabilities due to finite resolution--we just don't care to specify what the resolution is.
        analog31 4y ago
        I once had a TA job for an undergrad stats course. This was the "non calculus" course for the psych majors. I had also taken the "math" version of the same course, where we spent two semesters and proved everything. I honestly never came up with a satisfactory layman's explanation why continuous distributions are necessary, or what "continuous" is. I knew that we used calculus to derive the formulas that they were faced with memorizing, but that would have been irrelevant to them. The best explanation I can think of today is: Use the one that makes the math easier or more readable.
        DiogenesKynikos 4y ago
        Many measurements are continuous. What is the age of a rock? That's not a discrete quantity: it could be anything.
        analog31 4y ago
        Plus or minus what? I know the importance of continuous sets in the study of statistics as a branch of math. But I don't know of any measurements that can't be represented by integer multiples of a unit for all practical purposes. And the students in the non-calculus stats course can't grasp what continuity is anyway.
  - melling 4y ago
    I thought likelihood doesn’t necessarily sum to one so it’s not a probability.
    - MontyCarloHall 4y ago
      It will always sum (or integrate) to one with respect to the data. For example, given likelihood p(x1, x2, ..., x_N|params), summing (or integrating) over all possible values of x_1 ... x_N will indeed yield 1.
- MontyCarloHall 4y ago
  Likelihood: given a probabilistic model and its parameters, what is the probability of observing some data under that model? For example, given a coin with some probability f of getting heads and thus 1-f of getting tails (model parameter), a set of coin flips (data), and the assumption that each coin flip is independent of all others (model), the probability of seeing an arbitrary sequence with H total heads and T total tails after H+T flips is p(H,T|f) = f^H(1-f)^T [0] Note that this likelihood is normalized (i.e. sums up to 1) with respect to all possible sequences of flips of length H+T, e.g. for length 2, there are 4 possible sequences of coin flips: p(hh|f) + p(ht|f) + p(th|f) + p(tt|f) = f^2 + 2f(1-f) + (1-f)^2 = 1 (Also note that the likelihood is only a probability for discrete data; it’s a probability density for continuous data, since the underlying terms would no longer be probabilities but rather densities. In that case, replace the previous sum with an integral ranging over the entire domain of your continuous data.) Partition function (usually called a "marginal likelihood"): what if we instead normalize the likelihood with respect to the model parameters? Then it would no longer be a probability distribution with respect to the data, but rather a probability distribution with respect to the parameters. The marginal likelihood is just this normalizing constant. In the previous example, p(f|H,T) = p(H,T|f)/<constant that would normalize p(H,T|f) to integrate to 1 over all possible values of f> You can optionally weight this likelihood by a prior p(f), in which case the numerator would be p(H,T|f)p(f), with the partition function updated accordingly. >Also the above reveal there is more elaborate structure in bayes than just the idea of 'updating your prior with new information', which is simply a platitude among STEM folk I could not agree more. The core philosophical tenet of Bayesian inference is that this re-normalization of the likelihood returns something probabilistically meaningful. The debate over the validity of priors pales in comparison to the debate over whether the likelihood ought to be treated as proportional to a probability distribution over the model parameters. [0] Note that this is distinct from the probability of seeing any sequence with H heads and T tails. That would require normalizing with respect to the number of total sequences with H heads and T tails (H+T choose H), yielding the binomial distribution.
- oxff 4y ago
  The 'hardest' part is realizing that basically everyone reasons this way unless taught otherwise. The books do a lot of hard work to cover this up.
- jltsiren 4y ago
  The hard part is that the intuitive understanding of probability you learned as a kid is almost certainly wrong. You need to unlearn it and replace it with proper axiomatic treatment of probability based on measure theory. Once you have done that and built a new intuition, Bayesian thinking should feel pretty natural. Once you accept that probability is just the "size" of a set relative to the size of a superset and it's up to you to attach meaning to the sets, much of the confusion goes away.
  - nerdponx 4y ago
    I think you can get most of the way there with straightforward geometric intuition, treating the sample space as a rectangle of area 1 and proceeding from there into joint and conditional probability.
- lisper 4y ago
  No, the hardest part is realizing that there are all kinds of tacit assumptions that we bring to bear merely by formulating a problem for Bayesian analysis. Take the example used in chapter 2 of the book. It assumes that news articles can be classified as "real" or "fake", and there is no middle ground. It assumes that the initial prior produced by experts is reliable. And most of all it assumes that certain features, like exclamation points in the title, are causally related to realness or fakeness. To illustrate this last point, consider analyzing the titles for a different features, the presence of the letter "z" rather than the presence of exclamation points. If it turned out that fake news in the corpus used to generate the priors just turned out to have more zees in their titles, would you then be justified in concluding that a news article about Zanzibar was more likely to be fake because it contained two zees? That example might seem contrived, but if you use a Bayesian spam filter it is actually plausible that the presence of the letter "v" is dignostic of spam because of the prevalence of spam concerning viagra. But again, there is a causal model of this: viagra is a product that is often the subject of spam marketing, and the word "viagra" happens to have a v in it, which is a priori an uncommon letter in English. But all this can fall apart depending on the circumstances. If you one day joined an email discussion of Stradavarius violins, the v signal could suddenly fail. An even more dramatic example: suppose you are an academic who starts to do research on spam filters and you have a collaborator who starts to send you examples of hard-to-filter spam. Now you have a very strong signal, but no straightforward textual analysis will allow you to extract it. The hard part of Bayesian analysis is deciding what features to even look at. All the rest is borderline trivial by comparison.
  - bumby 4y ago
    >the hardest part is realizing that there are all kinds of tacit assumptions that we bring to bear merely by formulating a problem for Bayesian analysis. I think the common argument is this is a strength of Bayesian analysis. Namely, that your priors make you explicitly state your assumptions. All models integrate assumptions, but not all of them make you explicitly quantify them like Bayesian analysis does.
    - nerdponx 4y ago
- deleted 4y ago
  [deleted]
- analog31 4y ago
  >>> ... the idea of 'updating your prior with new information' ... ... dates back to antiquity. I'm not a statistician, but my impression is that Bayesian methods are a way of formalizing that approach. On the other hand, use of the term "prior" is somewhat misleading because the formulas do not specify a time sequence for acquiring information. I tend to have a lesser view of sprinkling Bayesian terminology into blogs about social issues. That's what I call Bayes Theater.
deleted 4y ago
[deleted]
_plg_ 4y ago
Review by Christian Robert, a statistician: https://xianblog.wordpress.com/2022/07/05/bayes-rules-book-review/ https://xianblog.wordpress.com/2022/07/05/bayes-rules-book-r...
- nerdponx 4y ago
  This is a really useful book review, thank you for sharing it (and thank you to the author for writing it!).
master_yoda_1 4y ago
I always get confused who the audience is for these kind of books? For advanced reader this is really shallow, for beginners its really advanced (who really want to read 543 page book as beginner).
- siddboots 4y ago
  Simply put, there are many people who are neither "beginners" nor "advanced". There is a section of the book titled Audience which reads > Bayes Rules! brings the power of Bayes to advanced undergraduate students and comparably trained practitioners. Accordingly, the book is neither written at the graduate level nor is it meant to be a first introduction to the field of statistics.
turbocon 4y ago
I've come across this format a couple times now, does anyone know how to export it to PDF? I honestly don't even mind paying for them but I want the flat file.
- t-3 4y ago
  Pandoc + wget wget -r https://www.bayesrulesbook.com/ https://www.bayesrulesbook.com/ && cd www.bayesrulesbook.com && pandoc --pdf-engine=xelatex foreword.html preface.html chapter-{1..19}.html references.html -o book.pdf
  - mkl 4y ago
    For me that repeats the table of contents many times, and leaves the maths as Latex code instead of formatting it as formulas.

147 points by RafelMri 4y ago

The scope seems pretty standard but I really like strongly structured form and exercises. Is there a pdf version available perchance?

platz 4y ago

The hardest part of bayes is 1) understanding what 'likelihood' actually is or represents 2) understanding what a 'partition function' actually is or represents I always forget what these mean since I don't use bayes in anger Also the above reveal there is more elaborate structure in bayes than just the idea of 'updating your prior with new information', which is simply a platitude among STEM folk

DiogenesKynikos 4y ago
> understanding what 'likelihood' actually is or represents The probability of observing the data you observed, assuming that the model parameters take on a certain value. > understanding what a 'partition function' actually is or represents The probability of observing the data you observed, this time averaging over all possible parameter values (weighted by the priors).
- psi75 4y ago
  > The probability of observing the data you observed Yes, in discrete cases. In continuous cases, you have to work with a probability density. I think this is one of the hurdles people encounter when they're first exposed to Bayesian stats. The probabilities, technically speaking, are zero. The important insight in Bayesian work is that it's often not the probabilities themselves that matter but the ratios thereof, since from those alone you can compute posteriors.
  - DiogenesKynikos 4y ago
    Indeed, but it's a bit tedious to always say, "probabilities (in the discrete case) or probability densities (in the continuous case)." In general, you have sums in the discrete case and integrals in the continuous case, but most formulas are otherwise the same.
    - psi75 4y ago
      That's quite true. Also, one could argue that continuous probabilities in practice are discrete probabilities due to finite resolution--we just don't care to specify what the resolution is.
      - analog31 4y ago
        I once had a TA job for an undergrad stats course. This was the "non calculus" course for the psych majors. I had also taken the "math" version of the same course, where we spent two semesters and proved everything. I honestly never came up with a satisfactory layman's explanation why continuous distributions are necessary, or what "continuous" is. I knew that we used calculus to derive the formulas that they were faced with memorizing, but that would have been irrelevant to them. The best explanation I can think of today is: Use the one that makes the math easier or more readable.
        DiogenesKynikos 4y ago
        Many measurements are continuous. What is the age of a rock? That's not a discrete quantity: it could be anything.
        analog31 4y ago
        Plus or minus what? I know the importance of continuous sets in the study of statistics as a branch of math. But I don't know of any measurements that can't be represented by integer multiples of a unit for all practical purposes. And the students in the non-calculus stats course can't grasp what continuity is anyway.
- melling 4y ago
  I thought likelihood doesn’t necessarily sum to one so it’s not a probability.
  - MontyCarloHall 4y ago
    It will always sum (or integrate) to one with respect to the data. For example, given likelihood p(x1, x2, ..., x_N|params), summing (or integrating) over all possible values of x_1 ... x_N will indeed yield 1.
MontyCarloHall 4y ago
Likelihood: given a probabilistic model and its parameters, what is the probability of observing some data under that model? For example, given a coin with some probability f of getting heads and thus 1-f of getting tails (model parameter), a set of coin flips (data), and the assumption that each coin flip is independent of all others (model), the probability of seeing an arbitrary sequence with H total heads and T total tails after H+T flips is p(H,T|f) = f^H(1-f)^T [0] Note that this likelihood is normalized (i.e. sums up to 1) with respect to all possible sequences of flips of length H+T, e.g. for length 2, there are 4 possible sequences of coin flips: p(hh|f) + p(ht|f) + p(th|f) + p(tt|f) = f^2 + 2f(1-f) + (1-f)^2 = 1 (Also note that the likelihood is only a probability for discrete data; it’s a probability density for continuous data, since the underlying terms would no longer be probabilities but rather densities. In that case, replace the previous sum with an integral ranging over the entire domain of your continuous data.) Partition function (usually called a "marginal likelihood"): what if we instead normalize the likelihood with respect to the model parameters? Then it would no longer be a probability distribution with respect to the data, but rather a probability distribution with respect to the parameters. The marginal likelihood is just this normalizing constant. In the previous example, p(f|H,T) = p(H,T|f)/<constant that would normalize p(H,T|f) to integrate to 1 over all possible values of f> You can optionally weight this likelihood by a prior p(f), in which case the numerator would be p(H,T|f)p(f), with the partition function updated accordingly. >Also the above reveal there is more elaborate structure in bayes than just the idea of 'updating your prior with new information', which is simply a platitude among STEM folk I could not agree more. The core philosophical tenet of Bayesian inference is that this re-normalization of the likelihood returns something probabilistically meaningful. The debate over the validity of priors pales in comparison to the debate over whether the likelihood ought to be treated as proportional to a probability distribution over the model parameters. [0] Note that this is distinct from the probability of seeing any sequence with H heads and T tails. That would require normalizing with respect to the number of total sequences with H heads and T tails (H+T choose H), yielding the binomial distribution.
oxff 4y ago
The 'hardest' part is realizing that basically everyone reasons this way unless taught otherwise. The books do a lot of hard work to cover this up.
jltsiren 4y ago
The hard part is that the intuitive understanding of probability you learned as a kid is almost certainly wrong. You need to unlearn it and replace it with proper axiomatic treatment of probability based on measure theory. Once you have done that and built a new intuition, Bayesian thinking should feel pretty natural. Once you accept that probability is just the "size" of a set relative to the size of a superset and it's up to you to attach meaning to the sets, much of the confusion goes away.
- nerdponx 4y ago
  I think you can get most of the way there with straightforward geometric intuition, treating the sample space as a rectangle of area 1 and proceeding from there into joint and conditional probability.
lisper 4y ago
No, the hardest part is realizing that there are all kinds of tacit assumptions that we bring to bear merely by formulating a problem for Bayesian analysis. Take the example used in chapter 2 of the book. It assumes that news articles can be classified as "real" or "fake", and there is no middle ground. It assumes that the initial prior produced by experts is reliable. And most of all it assumes that certain features, like exclamation points in the title, are causally related to realness or fakeness. To illustrate this last point, consider analyzing the titles for a different features, the presence of the letter "z" rather than the presence of exclamation points. If it turned out that fake news in the corpus used to generate the priors just turned out to have more zees in their titles, would you then be justified in concluding that a news article about Zanzibar was more likely to be fake because it contained two zees? That example might seem contrived, but if you use a Bayesian spam filter it is actually plausible that the presence of the letter "v" is dignostic of spam because of the prevalence of spam concerning viagra. But again, there is a causal model of this: viagra is a product that is often the subject of spam marketing, and the word "viagra" happens to have a v in it, which is a priori an uncommon letter in English. But all this can fall apart depending on the circumstances. If you one day joined an email discussion of Stradavarius violins, the v signal could suddenly fail. An even more dramatic example: suppose you are an academic who starts to do research on spam filters and you have a collaborator who starts to send you examples of hard-to-filter spam. Now you have a very strong signal, but no straightforward textual analysis will allow you to extract it. The hard part of Bayesian analysis is deciding what features to even look at. All the rest is borderline trivial by comparison.
- bumby 4y ago
  >the hardest part is realizing that there are all kinds of tacit assumptions that we bring to bear merely by formulating a problem for Bayesian analysis. I think the common argument is this is a strength of Bayesian analysis. Namely, that your priors make you explicitly state your assumptions. All models integrate assumptions, but not all of them make you explicitly quantify them like Bayesian analysis does.
  - nerdponx 4y ago
deleted 4y ago
[deleted]
analog31 4y ago
>>> ... the idea of 'updating your prior with new information' ... ... dates back to antiquity. I'm not a statistician, but my impression is that Bayesian methods are a way of formalizing that approach. On the other hand, use of the term "prior" is somewhat misleading because the formulas do not specify a time sequence for acquiring information. I tend to have a lesser view of sprinkling Bayesian terminology into blogs about social issues. That's what I call Bayes Theater.

deleted 4y ago

[deleted]

_plg_ 4y ago

Review by Christian Robert, a statistician: https://xianblog.wordpress.com/2022/07/05/bayes-rules-book-review/ https://xianblog.wordpress.com/2022/07/05/bayes-rules-book-r...

nerdponx 4y ago
This is a really useful book review, thank you for sharing it (and thank you to the author for writing it!).

master_yoda_1 4y ago

I always get confused who the audience is for these kind of books? For advanced reader this is really shallow, for beginners its really advanced (who really want to read 543 page book as beginner).

siddboots 4y ago
Simply put, there are many people who are neither "beginners" nor "advanced". There is a section of the book titled Audience which reads > Bayes Rules! brings the power of Bayes to advanced undergraduate students and comparably trained practitioners. Accordingly, the book is neither written at the graduate level nor is it meant to be a first introduction to the field of statistics.

turbocon 4y ago

I've come across this format a couple times now, does anyone know how to export it to PDF? I honestly don't even mind paying for them but I want the flat file.

t-3 4y ago
Pandoc + wget wget -r https://www.bayesrulesbook.com/ https://www.bayesrulesbook.com/ && cd www.bayesrulesbook.com && pandoc --pdf-engine=xelatex foreword.html preface.html chapter-{1..19}.html references.html -o book.pdf
- mkl 4y ago
  For me that repeats the table of contents many times, and leaves the maths as Latex code instead of formatting it as formulas.