10 ms·

Could somebody explain what is Bayesian analysis and how it works or point to an article explaining it ? The wikipedia article regarding Bayesian analysis is c

Could somebody explain what is Bayesian analysis and how it works or point to an article explaining it ? The wikipedia article regarding Bayesian analysis is cryptic to me.

jules 16y ago
Here's a wikipedia page with an example http://en.wikipedia.org/wiki/Checking_whether_a_coin_is_fair http://en.wikipedia.org/wiki/Checking_whether_a_coin_is_fair The problem is as follows. I give you a sequence of coin flips, for example TTTHHTTT, and your task is to determine whether this coin is fair. Obviously you can't answer this question with certainty. The frequentist approach seems rather ridiculous to me. They pretend to be able to answer this question without knowing anything about coins. Instead of making the assumptions up front, they hide the assumptions in the method of determining whether the coin is fair. For example one would think that to decide whether a coin I give you is fair, you'd need some idea about what kind of coins I will give you. If we were talking about reality, you would be able to say "no it's not fair" without even looking at the data sequence, because in reality there are no fair coins. If we are in a different universe where fair coins do exist, you'd need some idea how many are fair. So obviously whether a coin is fair depends on which universe you live in, but the frequentist method is not parameterized by the universe. The assumptions are implicitly made inside the method. The bayesian approach asks you first to state your prior ideas about coins. Then you give the bayesian your data, and he will compute for you the probability that this coin is fair. This cleanly separates the correct mathematical derivation from the subjective assumptions, instead of hiding these assumptions inside the method. Moreover, the bayesian approach is automatic, in principle. Once you make your assumptions clear, the rest is just mechanical derivation. The frequentist approach requires divine inspiration, and is very easy to get wrong. For example you're not allowed to look at the data before formulating your hypothesis. Of course nobody does this right in practice. I've often seen people cherry pick a frequentist statistical test that proves their hypothesis. Given any data and any hypothesis, I can devise a correct frequentist statistical test that proves the hypothesis with arbitrarily large confidence.
- hessenwolf 16y ago
  Would that the world were about flipping coins. What's your prior distribution on the volatility parameter of a model of the 3-month implied volatility of the S&P 500 stock index? Now, justify it. What is your prior distribution on the rate at which bugs are found in a software system that runs a nuclear plant? Now, justify it. Personally, my priors are usually crap.
  - jules 16y ago
    I agree. But with frequentist methods it's not even clear what kind of assumptions you're making. Explicitly stating your assumptions is better than implicitly hiding them in your method. "We can see that the bayesian approach won't work well here, so lets use frequentist methods because we cannot see that they won't work well." The problem with the nuclear plant is that we don't have much data on nuclear plant software bugs. However you could still get an idea by using bug rates in other software as your priors, and acknowledge that your priors are not perfect. OR you could use frequentist methods to make up for the lack of data, and know you will be able to devise your method in such a way that it proves your hypothesis. Or if you use the method correctly (that is don't look at your data, and pick a statistical test beforehand) make no useful predictions at all.
    - hackerblues 16y ago
      "But with frequentist methods it's not even clear what kind of assumptions you're making." I don't understand what you mean by this. For a simple, concrete example: take the problem of fitting a distribution to a sample of real random variables. It seems that a Frequentist would make the following assumptions: 1. The data comes iid from some unknown but fixed distribution. 2. This true distribution is included within some set of distributions: eg, the set of normal distributions with real mean and non-negative real variance. They would then use some estimator to estimate the parameters. The choice of estimator is perhaps justified by some theoretical properties, eg, consistency etc. This gives another assumption. 3. The best estimator to use is the one which is optimal with regard to properties x,y,z. If it is not clear what assumptions I am making and I have explicitly stated some, presumably there are others which I have left unstated. Could you explain why you think the above collection is insufficient?
sharpneli 16y ago
The highly recommended classic: http://yudkowsky.net/rational/bayes http://yudkowsky.net/rational/bayes
- deskamess 16y ago
  I could not solve the following problem from the article. Suppose that 40% of the eggs are painted blue, 5/13 of the eggs containing pearls are painted blue, and 20% of the eggs are both empty and painted red. What is the probability that an egg painted blue contains a pearl? Any hints/ideas?
  - tel 16y ago
    It requires a little bit of indirection. You'll find that you could easily solve the equation if you only knew one more quantity. So call the unknown "X" and keep going. You'll be able to form an equation that constrains X to a single value. FWIW, the answer I get has three non-zero decimal digits.
  - defen 16y ago
    Pr(pearl|blue) = Pr(blue|pearl) * Pr(pearl) / Pr(blue) Two are given outright, and you should be able to compute the third based on the available data.
  - jules 16y ago
    Make a table: | blue | red | --------|------|------|----- pearls | | | --------|------|------|----- empty | | | --------|------|------|----- | | | 100 We know that 40% of the eggs are blue, so we fill in 40 at the bottom row in the blue column. We now also know that 60% are red. Similarly, 20% of eggs are empty & red, so we fill in 20 there. Now we can calculate that 60% - 20% = 40% are pearls & red: | blue | red | --------|------|------|----- pearls | | 40 | --------|------|------|----- empty | | 20 | --------|------|------|----- | 40 | 60 | 100 Now comes the final bit of data: 5/13 of the eggs containing pearls are painted blue: | blue | red | --------|------|------|----- pearls | X | 40 | 40+X --------|------|------|----- empty | | 20 | --------|------|------|----- | 40 | 60 | 100 With X = 5/13*(40+X), or X = 100/3. | blue | red | --------|-------|------|----- pearls | 100/3 | 40 | 220/3 --------|-------|------|----- empty | 20/3 | 20 | 80/3 --------|-------|------|----- | 40 | 60 | 100 Now we can answer the question: what is the probability that an egg painted blue contains a pearl? For every 40 blue eggs, 100/3 contain pearls. So the answer is 100/3/40 = 5/6.
hessenwolf 16y ago
Frequentist Statistics: 1. Start with a model that you think describes your data. 2. Find the parameters of the model that make it fit the observed data most closely. 3. If the model really does not seem to fit the parameters, reject the model. The criteria for it not to fit is usually the p-value, which is the probability of data at least as unlikely as your observations occurring if your model is actually correct. If the p-value is less than .05 (by an arbitrary and accidental custom) the model is rejected. Wrap your head around that, if you dare. 4. If the model is not rejected, use it to predict future outcomes with the best-fit assumptions, ideally allowing for some uncertainty in the values of the parameters. Decisions are made on this basis. The uncertainty in the values of the parameters is quite difficult to allow for in practice. Bayesian Statistics. 1. Start with a model that you think describes the data. 2. Then get some models to model the parameters of your model, called prior models, describing your uncertainty about the parameters of your model. You get the initial parameters for the prior models from your own head, or the head of an field expert. 3. Use the observed data to refine your estimates of the parameters for the prior models. So you started with educated guesses of these 'prior' parameters, and then you made the guesses better using your observations. In practice, your guesses can become irrelevant very quickly as you add more data. 4. Predict future outcomes using your model, where the uncertainty in the model parameters is modelled explicitly using your prior models as described in 3. 5. Put those predictions of future outcomes into a utility function to make decisions. Summary: The main thing is that Bayesian statistics allows you to specify models for your parameter uncertainty, provided you are okay with the educated guesses.
- _delirium 16y ago
  You seem to be conflating quite a few axes of variation. What you're describing as "frequentist" is descriptive statistics using parametric models, which is only one possible way of using a frequentist interpretation of probability. Nonparametric statistics is probably the biggest active area of frequentist-statistics research, in which case fitting parameters of models isn't exactly what you're doing (though there is some sort of model-building process, and some arguing about what constitutes a parameter). In addition, predictive statistics is a quite large area of frequentist statistics, and it does precisely what you call "Bayesian" steps #4 and #5, except within a frequentist framework: you fit a predictive model, which may include uncertainty estimates in its predictions, and then feed that model's output through something decision-theoretic, like a risk function. Bayesian decision theory and frequentist decision theory do look different, but it's not as if frequentists don't have a decision theory (and a real one, not just "use the model if it has a low enough p-value")...

Could somebody explain what is Bayesian analysis and how it works or point to an article explaining it ? The wikipedia article regarding Bayesian analysis is cryptic to me.

jules 16y ago

Here's a wikipedia page with an example http://en.wikipedia.org/wiki/Checking_whether_a_coin_is_fair http://en.wikipedia.org/wiki/Checking_whether_a_coin_is_fair The problem is as follows. I give you a sequence of coin flips, for example TTTHHTTT, and your task is to determine whether this coin is fair. Obviously you can't answer this question with certainty. The frequentist approach seems rather ridiculous to me. They pretend to be able to answer this question without knowing anything about coins. Instead of making the assumptions up front, they hide the assumptions in the method of determining whether the coin is fair. For example one would think that to decide whether a coin I give you is fair, you'd need some idea about what kind of coins I will give you. If we were talking about reality, you would be able to say "no it's not fair" without even looking at the data sequence, because in reality there are no fair coins. If we are in a different universe where fair coins do exist, you'd need some idea how many are fair. So obviously whether a coin is fair depends on which universe you live in, but the frequentist method is not parameterized by the universe. The assumptions are implicitly made inside the method. The bayesian approach asks you first to state your prior ideas about coins. Then you give the bayesian your data, and he will compute for you the probability that this coin is fair. This cleanly separates the correct mathematical derivation from the subjective assumptions, instead of hiding these assumptions inside the method. Moreover, the bayesian approach is automatic, in principle. Once you make your assumptions clear, the rest is just mechanical derivation. The frequentist approach requires divine inspiration, and is very easy to get wrong. For example you're not allowed to look at the data before formulating your hypothesis. Of course nobody does this right in practice. I've often seen people cherry pick a frequentist statistical test that proves their hypothesis. Given any data and any hypothesis, I can devise a correct frequentist statistical test that proves the hypothesis with arbitrarily large confidence.

hessenwolf 16y ago
Would that the world were about flipping coins. What's your prior distribution on the volatility parameter of a model of the 3-month implied volatility of the S&P 500 stock index? Now, justify it. What is your prior distribution on the rate at which bugs are found in a software system that runs a nuclear plant? Now, justify it. Personally, my priors are usually crap.
- jules 16y ago
  I agree. But with frequentist methods it's not even clear what kind of assumptions you're making. Explicitly stating your assumptions is better than implicitly hiding them in your method. "We can see that the bayesian approach won't work well here, so lets use frequentist methods because we cannot see that they won't work well." The problem with the nuclear plant is that we don't have much data on nuclear plant software bugs. However you could still get an idea by using bug rates in other software as your priors, and acknowledge that your priors are not perfect. OR you could use frequentist methods to make up for the lack of data, and know you will be able to devise your method in such a way that it proves your hypothesis. Or if you use the method correctly (that is don't look at your data, and pick a statistical test beforehand) make no useful predictions at all.
  - hackerblues 16y ago
    "But with frequentist methods it's not even clear what kind of assumptions you're making." I don't understand what you mean by this. For a simple, concrete example: take the problem of fitting a distribution to a sample of real random variables. It seems that a Frequentist would make the following assumptions: 1. The data comes iid from some unknown but fixed distribution. 2. This true distribution is included within some set of distributions: eg, the set of normal distributions with real mean and non-negative real variance. They would then use some estimator to estimate the parameters. The choice of estimator is perhaps justified by some theoretical properties, eg, consistency etc. This gives another assumption. 3. The best estimator to use is the one which is optimal with regard to properties x,y,z. If it is not clear what assumptions I am making and I have explicitly stated some, presumably there are others which I have left unstated. Could you explain why you think the above collection is insufficient?

sharpneli 16y ago

The highly recommended classic: http://yudkowsky.net/rational/bayes http://yudkowsky.net/rational/bayes

deskamess 16y ago
I could not solve the following problem from the article. Suppose that 40% of the eggs are painted blue, 5/13 of the eggs containing pearls are painted blue, and 20% of the eggs are both empty and painted red. What is the probability that an egg painted blue contains a pearl? Any hints/ideas?
- tel 16y ago
  It requires a little bit of indirection. You'll find that you could easily solve the equation if you only knew one more quantity. So call the unknown "X" and keep going. You'll be able to form an equation that constrains X to a single value. FWIW, the answer I get has three non-zero decimal digits.
- defen 16y ago
  Pr(pearl|blue) = Pr(blue|pearl) * Pr(pearl) / Pr(blue) Two are given outright, and you should be able to compute the third based on the available data.
- jules 16y ago
  Make a table: | blue | red | --------|------|------|----- pearls | | | --------|------|------|----- empty | | | --------|------|------|----- | | | 100 We know that 40% of the eggs are blue, so we fill in 40 at the bottom row in the blue column. We now also know that 60% are red. Similarly, 20% of eggs are empty & red, so we fill in 20 there. Now we can calculate that 60% - 20% = 40% are pearls & red: | blue | red | --------|------|------|----- pearls | | 40 | --------|------|------|----- empty | | 20 | --------|------|------|----- | 40 | 60 | 100 Now comes the final bit of data: 5/13 of the eggs containing pearls are painted blue: | blue | red | --------|------|------|----- pearls | X | 40 | 40+X --------|------|------|----- empty | | 20 | --------|------|------|----- | 40 | 60 | 100 With X = 5/13*(40+X), or X = 100/3. | blue | red | --------|-------|------|----- pearls | 100/3 | 40 | 220/3 --------|-------|------|----- empty | 20/3 | 20 | 80/3 --------|-------|------|----- | 40 | 60 | 100 Now we can answer the question: what is the probability that an egg painted blue contains a pearl? For every 40 blue eggs, 100/3 contain pearls. So the answer is 100/3/40 = 5/6.

hessenwolf 16y ago

Frequentist Statistics: 1. Start with a model that you think describes your data. 2. Find the parameters of the model that make it fit the observed data most closely. 3. If the model really does not seem to fit the parameters, reject the model. The criteria for it not to fit is usually the p-value, which is the probability of data at least as unlikely as your observations occurring if your model is actually correct. If the p-value is less than .05 (by an arbitrary and accidental custom) the model is rejected. Wrap your head around that, if you dare. 4. If the model is not rejected, use it to predict future outcomes with the best-fit assumptions, ideally allowing for some uncertainty in the values of the parameters. Decisions are made on this basis. The uncertainty in the values of the parameters is quite difficult to allow for in practice. Bayesian Statistics. 1. Start with a model that you think describes the data. 2. Then get some models to model the parameters of your model, called prior models, describing your uncertainty about the parameters of your model. You get the initial parameters for the prior models from your own head, or the head of an field expert. 3. Use the observed data to refine your estimates of the parameters for the prior models. So you started with educated guesses of these 'prior' parameters, and then you made the guesses better using your observations. In practice, your guesses can become irrelevant very quickly as you add more data. 4. Predict future outcomes using your model, where the uncertainty in the model parameters is modelled explicitly using your prior models as described in 3. 5. Put those predictions of future outcomes into a utility function to make decisions. Summary: The main thing is that Bayesian statistics allows you to specify models for your parameter uncertainty, provided you are okay with the educated guesses.

_delirium 16y ago
You seem to be conflating quite a few axes of variation. What you're describing as "frequentist" is descriptive statistics using parametric models, which is only one possible way of using a frequentist interpretation of probability. Nonparametric statistics is probably the biggest active area of frequentist-statistics research, in which case fitting parameters of models isn't exactly what you're doing (though there is some sort of model-building process, and some arguing about what constitutes a parameter). In addition, predictive statistics is a quite large area of frequentist statistics, and it does precisely what you call "Bayesian" steps #4 and #5, except within a frequentist framework: you fit a predictive model, which may include uncertainty estimates in its predictions, and then feed that model's output through something decision-theoretic, like a risk function. Bayesian decision theory and frequentist decision theory do look different, but it's not as if frequentists don't have a decision theory (and a real one, not just "use the model if it has a low enough p-value")...