3 ms·

What is the theoretical justification for taking a completely flat prior? "If we set the prior P(Ftrue)∝1 (a flat prior)," There's no probability distribution

What is the theoretical justification for taking a completely flat prior? "If we set the prior P(Ftrue)∝1 (a flat prior)," There's no probability distribution which is constant over the whole real line. Is the idea that we can pick a distribution which is constant over an arbitrarily large (but finite) interval around the observed data, and so in practice, we may get results arbitrarily close to those given?

jmount 12y ago
I'd say the justification is two parts. First is the Bernstein–von Mises theorem (priors don't matter once you have enough data, as long as you didn't violate Cromwell’s rule by using zeros). The second part is: improper priors are considered okay- as long as you check the posterior corresponds to a sensible distribution.
- Houshalter 12y ago
  If I understand correctly, assuming all real numbers have equal prior probability, then no matter how much data you gather, it's still infinitely unlikely that the true value will exist within any finite range. E.g. the probability of the true value being between -100! and +100! is 0. If you draw from the distribution you will always draw "infinities".
  - jmount 12y ago
    That is why you have to check. If the P(data|param) is concentrated, then even for uniform priors on the real like you can have P(param|data) is proportional to a sensible distribution. But you have to check for specific model and data (say P(data|param) = c e^{(data-param)^2}), it isn't enough to verbally work through infinities.
micro_cam 12y ago
Actually kind of the opposite idea. So called "uninformative priors" can be considered a form or model regularization in that they prevent the model from fitting the data too well...they spread the posterior distribution out. Weakly informative priors like a broad normal distribution are also popular and may yield better results.
pash 12y ago
Probabilists often think of a uniform distribution over the whole real line as giving an infinitesimal (but non-zero) probability at each point. The intuition is very similar to the way physicists understand the Dirac delta function. Like the Dirac delta, an improper prior doesn't have any formally acceptable definition as a stand-alone mathematical object [0], but it does have a good definition that covers the way it's used. Now, why does a uniform distribution correspond to an "uninformative" or "unbiased" prior in the first place? Because the uniform is the unconstrained maximum-entropy distribution. If you don't have rock-solid intuition about the concept of entropy, I recommend starting with Arieh Ben-Naim's book [1]. 0. But with nonstandard analysis, it's a simple thing to give a rigorous, intuitive definition of a uniform distribution whose support includes as much of the real line as we could possibly care about. 1. A Farewell to Entropy, http://www.amazon.com/FAREWELL-ENTROPY-Statistical-Thermodynamics-Information/dp/9812707069 http://www.amazon.com/FAREWELL-ENTROPY-Statistical-Thermodyn.... It's a probability book masquerading as a physics book.

jmount 12y ago

I'd say the justification is two parts. First is the Bernstein–von Mises theorem (priors don't matter once you have enough data, as long as you didn't violate Cromwell’s rule by using zeros). The second part is: improper priors are considered okay- as long as you check the posterior corresponds to a sensible distribution.

Houshalter 12y ago
If I understand correctly, assuming all real numbers have equal prior probability, then no matter how much data you gather, it's still infinitely unlikely that the true value will exist within any finite range. E.g. the probability of the true value being between -100! and +100! is 0. If you draw from the distribution you will always draw "infinities".
- jmount 12y ago
  That is why you have to check. If the P(data|param) is concentrated, then even for uniform priors on the real like you can have P(param|data) is proportional to a sensible distribution. But you have to check for specific model and data (say P(data|param) = c e^{(data-param)^2}), it isn't enough to verbally work through infinities.

micro_cam 12y ago

Actually kind of the opposite idea. So called "uninformative priors" can be considered a form or model regularization in that they prevent the model from fitting the data too well...they spread the posterior distribution out. Weakly informative priors like a broad normal distribution are also popular and may yield better results.

pash 12y ago

Probabilists often think of a uniform distribution over the whole real line as giving an infinitesimal (but non-zero) probability at each point. The intuition is very similar to the way physicists understand the Dirac delta function. Like the Dirac delta, an improper prior doesn't have any formally acceptable definition as a stand-alone mathematical object [0], but it does have a good definition that covers the way it's used. Now, why does a uniform distribution correspond to an "uninformative" or "unbiased" prior in the first place? Because the uniform is the unconstrained maximum-entropy distribution. If you don't have rock-solid intuition about the concept of entropy, I recommend starting with Arieh Ben-Naim's book [1]. 0. But with nonstandard analysis, it's a simple thing to give a rigorous, intuitive definition of a uniform distribution whose support includes as much of the real line as we could possibly care about. 1. A Farewell to Entropy, http://www.amazon.com/FAREWELL-ENTROPY-Statistical-Thermodynamics-Information/dp/9812707069 http://www.amazon.com/FAREWELL-ENTROPY-Statistical-Thermodyn.... It's a probability book masquerading as a physics book.