4 ms·

The well-calibrated Bayesian [pdf] (1982)

51 points by Murfalo 3mo ago

ckrapu 3mo ago
This is a classic!
lkbm 3mo ago
(1982)
HPsquared 3mo ago
Title a reference to The Well-Tempered Clavier?
- lioeters 3mo ago
  I'm sure it is, and thought it's a clever and humorous title. It made me wonder, what other books and articles on scientific or mathematical subjects reference works of classical music, art, literature. Goedel Escher Bach, of course, but now I'm curious to find more of such interdisciplinary works.
wosk 3mo ago
It took me a while, but here’s what I gather about this (I’m pretty sure it’s correct, but I’m not an expert). A calibrated forecast means that if you say there is a 20% chance of rain, then it actually rains 20% of the time. It’s a desired feature, but not the only one (e.g. you could be calibrated by stating: Chick-fil-A is open every day except Monday, but your forecast will always be wrong on Sunday and Monday). So if 1. you are Bayesian (you state your beliefs) 2. and coherent (the laws of probability apply, so e.g. if P(A) = 0.4, then P(not A) cannot be anything other than 0.6): and you are predicting something (e.g rain tomorrow), then if you believe it will rain with probability 0.7 but you are 80% sure of your belief, you won’t say 0.7; you will say something else: 0.8 × 0.7 + 0.2 × something_you_believe = 0.58. Coherence forces you to collapse your uncertainty into your probability at each forecast. This theorem shows that, over many forecasts, in your belief system you are certain to be producing a calibrated forecast: your current beliefs assign probability 1 to the proposition that your future forecasts will be calibrated. But that can’t be, which is the paradox. So Bayesianism is too strong compared to how scientists reason, because scientists always think their model can have an error.
- PaulHoule 3mo ago
  I didn't see a real Bayesian point of view in that article. A Bayesian does not give you a probability estimate they give you a probability distribution for the probability! Like in Star Trek Spock is always saying something like "Captain, we have a 15.31% chance of surviving this mission" which is a ridiculous example of precision without accuracy. [1] If you observe a coin flipped 100 times and it came up heads 65 times it is not a crazy point estimate to say it has a 65% chance of coming up heads but this is just one sample and if you did it another time maybe it comes up 61 or 68 times. You are better saying that the probability distribution of the probability is β(65,35) or maybe β(65.5,35.5) or β(66,36) since that has the "error bars" built in, can be updated if you get more samples, etc. [1] ... and you know he underestimates survival probabilities the same way Scotty overestimates how long it will take to fix the engines
  - clickety_clack 3mo ago
    Was Spock poorly calibrated?
    - PaulHoule 3mo ago
      If he was well calibrated there is no way they would have made it through 79 episodes!
      - bryanrasmussen 3mo ago
        Spock had not realized that James Kirk emitted a psionic reality distortion field through higher dimensional "luck", if he had he would have been a bit more relaxed.
        oliculipolicula 3mo ago
        To his defence. in the trekkie timeline, neither Vulcans nor humans had discovered the Metropolis algorithm, so Spock had (for sure) nothing with which to calibrate that
dgritsko 3mo ago
Must ensure that this quote is present whenever Cromwell's Rule is mentioned - "I beseech you, in the bowels of Christ, think it possible that you may be mistaken."
deleted 3mo ago
[deleted]
danbruc 3mo ago
Something Bayesian. Despite my best effort I just do not get Bayesian probability, it more or less just does not make sense to me. Can you convince me otherwise? What is your best example of something with a probability that can not be analyzed in terms of frequencies or other proportions? And your Bayesian account of it must make sense, I am 90 % certain that P != NP and that is why I would take bets based on those odds does not cut it.
- jonahx 3mo ago
  Any one off event is an example. But I assume you know that, so can you clarify what you mean by "a probability that can not be analyzed in terms of frequencies or other proportions"?
  - danbruc 3mo ago
    Let us start with coin flips. You repeatedly flip a coin and the number of heads will come out to be about half the number of trials. Where does that come from? It is not some intrinsic property of the coin, it comes from varying initial conditions. If you had enough precision when controlling your hand movements, you could in principle force an outcome with high probability. But assuming you can not or at least do not do that, there is a certain set of initial states, some will lead to heads, some to tails, and each toss will start from a randomly selected initial state. So given my ignorance of the exact initial state, the coin will land heads with a probability equal to the number of initial states leading to heads divided by the number of initial states compatible with my observations of the initial state. [1] Repeatedly tossing a coin will sample the set of initial states and the result will match the proportion of the number of states. At least as long as I am not wrong about the set of initial states. The same applies to something like an election. I have imperfect knowledge about the state of the world but there is a set of states compatible with my knowledge about the world and certain subsets of them will lead to certain candidates to win. [1] Maybe adjusted by some probability distribution over the initial states if they are not equally likely to be picked.
- kgwgk 3mo ago
  What's the probability that the sinking of the USS Maine in 1898 was accidental?
arpi09 3mo ago
[flagged]

51 points by Murfalo 3mo ago

ckrapu 3mo ago

This is a classic!

lkbm 3mo ago

(1982)

HPsquared 3mo ago

Title a reference to The Well-Tempered Clavier?

lioeters 3mo ago
I'm sure it is, and thought it's a clever and humorous title. It made me wonder, what other books and articles on scientific or mathematical subjects reference works of classical music, art, literature. Goedel Escher Bach, of course, but now I'm curious to find more of such interdisciplinary works.

wosk 3mo ago

It took me a while, but here’s what I gather about this (I’m pretty sure it’s correct, but I’m not an expert). A calibrated forecast means that if you say there is a 20% chance of rain, then it actually rains 20% of the time. It’s a desired feature, but not the only one (e.g. you could be calibrated by stating: Chick-fil-A is open every day except Monday, but your forecast will always be wrong on Sunday and Monday). So if 1. you are Bayesian (you state your beliefs) 2. and coherent (the laws of probability apply, so e.g. if P(A) = 0.4, then P(not A) cannot be anything other than 0.6): and you are predicting something (e.g rain tomorrow), then if you believe it will rain with probability 0.7 but you are 80% sure of your belief, you won’t say 0.7; you will say something else: 0.8 × 0.7 + 0.2 × something_you_believe = 0.58. Coherence forces you to collapse your uncertainty into your probability at each forecast. This theorem shows that, over many forecasts, in your belief system you are certain to be producing a calibrated forecast: your current beliefs assign probability 1 to the proposition that your future forecasts will be calibrated. But that can’t be, which is the paradox. So Bayesianism is too strong compared to how scientists reason, because scientists always think their model can have an error.

PaulHoule 3mo ago
I didn't see a real Bayesian point of view in that article. A Bayesian does not give you a probability estimate they give you a probability distribution for the probability! Like in Star Trek Spock is always saying something like "Captain, we have a 15.31% chance of surviving this mission" which is a ridiculous example of precision without accuracy. [1] If you observe a coin flipped 100 times and it came up heads 65 times it is not a crazy point estimate to say it has a 65% chance of coming up heads but this is just one sample and if you did it another time maybe it comes up 61 or 68 times. You are better saying that the probability distribution of the probability is β(65,35) or maybe β(65.5,35.5) or β(66,36) since that has the "error bars" built in, can be updated if you get more samples, etc. [1] ... and you know he underestimates survival probabilities the same way Scotty overestimates how long it will take to fix the engines
- clickety_clack 3mo ago
  Was Spock poorly calibrated?
  - PaulHoule 3mo ago
    If he was well calibrated there is no way they would have made it through 79 episodes!
    - bryanrasmussen 3mo ago
      Spock had not realized that James Kirk emitted a psionic reality distortion field through higher dimensional "luck", if he had he would have been a bit more relaxed.
      - oliculipolicula 3mo ago
        To his defence. in the trekkie timeline, neither Vulcans nor humans had discovered the Metropolis algorithm, so Spock had (for sure) nothing with which to calibrate that

dgritsko 3mo ago

Must ensure that this quote is present whenever Cromwell's Rule is mentioned - "I beseech you, in the bowels of Christ, think it possible that you may be mistaken."

deleted 3mo ago

[deleted]

danbruc 3mo ago

Something Bayesian. Despite my best effort I just do not get Bayesian probability, it more or less just does not make sense to me. Can you convince me otherwise? What is your best example of something with a probability that can not be analyzed in terms of frequencies or other proportions? And your Bayesian account of it must make sense, I am 90 % certain that P != NP and that is why I would take bets based on those odds does not cut it.

jonahx 3mo ago
Any one off event is an example. But I assume you know that, so can you clarify what you mean by "a probability that can not be analyzed in terms of frequencies or other proportions"?
- danbruc 3mo ago
  Let us start with coin flips. You repeatedly flip a coin and the number of heads will come out to be about half the number of trials. Where does that come from? It is not some intrinsic property of the coin, it comes from varying initial conditions. If you had enough precision when controlling your hand movements, you could in principle force an outcome with high probability. But assuming you can not or at least do not do that, there is a certain set of initial states, some will lead to heads, some to tails, and each toss will start from a randomly selected initial state. So given my ignorance of the exact initial state, the coin will land heads with a probability equal to the number of initial states leading to heads divided by the number of initial states compatible with my observations of the initial state. [1] Repeatedly tossing a coin will sample the set of initial states and the result will match the proportion of the number of states. At least as long as I am not wrong about the set of initial states. The same applies to something like an election. I have imperfect knowledge about the state of the world but there is a set of states compatible with my knowledge about the world and certain subsets of them will lead to certain candidates to win. [1] Maybe adjusted by some probability distribution over the initial states if they are not equally likely to be picked.
kgwgk 3mo ago
What's the probability that the sinking of the USS Maine in 1898 was accidental?

arpi09 3mo ago

[flagged]