5 ms·

And your Bayesian statistician will acquire their priors how exactly? Bayesian statistics can be very powerful, but it would be a terrible idea to prefer Bayes

by pja 13y ago

And your Bayesian statistician will acquire their priors how exactly? Bayesian statistics can be very powerful, but it would be a terrible idea to prefer Bayesian approaches to Frequentist ones in all situations.

loup-vaillant 13y ago
> And your Bayesian statistician will acquire their priors how exactly? As If Frequentists somehow didn't need priors. Everyone starts with prior knowledge. We might as well use it. Or do you advocate not using every scrap of knowledge available to you? That would be stupid. Sure, prior knowledge can be shaky, or difficult to justify. But at least, a Bayesian will be explicit about it, instead of, like, sweeping normal probability distribution assumptions under the linear regression rug. > it would be a terrible idea to prefer Bayesian approaches to Frequentist ones in all situations. Name three examples that doesn't involve the Frequentist using better prior information than the Bayesian. By the way, Bayesians know that using probability theory correctly is sometimes intractable (combinatorial explosion and all that). In those cases, they will use approximations. But at least, they will know it's an approximation. --- You really should read chapters 1 and 2 of Probability Theory: the Logic of Science. They give a good feel of why Bayesians are correct as a simple matter of fact.
- rfergie 13y ago
  Those chapters show that any consistent (in some sense) statistical reasoning system must result in the same answers as a Bayesian system. They don't show that the frequentist approach is wrong. If both methods result in the same answer and frequentist methods are easier to use then what is the problem?
  - loup-vaillant 13y ago
    No problem of course. But that's a big if. And how are you going to evaluate whether both methods do in fact deliver the same answer? Also, the Frequentist approach is not wrong. It's inaccurate, to the extent its results differ from the Bayesian ones. This inaccuracy tend to go down as we gather more data. Which is a good thing, or else science itself wouldn't work. One more thing. You said "in some sense". Are you seriously suggesting that the assumptions behind Cox's Theorem can reasonably be challenged?
    - thebooktocome 13y ago
      > Are you seriously suggesting that the assumptions behind Cox's Theorem can reasonably be challenged? Sure: probability is continuous.
      - loup-vaillant 13y ago
        You mean, "it is not totally obvious that probability is continuous". Well… To me, it is obvious.
    - rfergie 13y ago
      I'm ashamed to say that I said "in some sense" to be deliberately vague about what I meant by consistency - I didn't want to overstate the strength of Cox's theorem or end up arguing about Godel.
      - loup-vaillant 13y ago
        Actually, I was accusing you of understating the strength of Cox's theorem.
- jtleek 13y ago
  As the author of the guide in question, I feel I should speak up here. Neither frequentist or Bayesian statistics is "just right". It is 100% dependent on the user and whether they use/interpret the quantities correctly. There are both really good and horrible Frequentist and Bayesian statisticians. To imply one or the other is "better" is incorrect and disingenuous. Just my 2 cents.
  - loup-vaillant 13y ago
    Oops. It seems I got carried away. > There are both really good and horrible Frequentist and Bayesian statisticians. Yeah. If I had to choose between Fisher and Anonymous Bayesian, I may chose Fisher. However, unless both kind of statistics yield the same results (I don't think they do), then at least one of them is bogus, by application of the non-contradiction principle. So, while I can imagine there are good Frequentists Statisticians out there, I insist that frequentism itself is bogus.
- dspeyer 13y ago
  Here's two examples: I'm testing the effectiveness of a drug. Drugs of this class have a certain likelihood of working, the noise in my data is known, the experimental group did this much better than the control... does the drug really work? So far so trivial, in either Bayesianism or Frequentism. Now, I happen to mention that I tested 10000 variants of this drug and only sent data for the one that seemed to work. The rest aren't interesting after all. Under Frequentism, it's easy to take this into account. Under Bayesianism, it requires complex definitions of observations, and is easy to overlook as there's no space for it in the formula. I have a collection of unfair dice. Unfortunately, they all look the same and got dumped on the floor. Now someone grabbed one off the floor at random and wants to make bets with me about it. Even experienced Bayesians are likely to mix up their propositions in a case like this. I say that from having read discussions of similar problems. Yes, if you do it right, it comes out correctly, but Frequentism makes sure you've thought about what you're asking in the same way Bayesianism makes sure you've thought about your priors. Somebody else will have to give a third example. Bayesianism and Frequentism are based on the same math, and math is math. If you use them correctly, they'll get you the same answer every time. The difference is what they make easy, and what mistakes they protect you against.
  - loup-vaillant 13y ago
    First example: of course it's complex (though I have no idea what you mean by "definitions of observations"). The correct answer needs my probability distribution over the algorithm your brain used to select which drug to send to me (my brain already hurts). Also, could you describe the Frequentist method in more detail? I'm sure it must overlook something. Second example: Okay, Bayesian statistics are harder. That's a disadvantage. --- > If you use [Bayesianism or Frequentism] correctly, they'll get you the same answer every time Wat. If they gave invariably the same answer, then, why the endless debates? By the way, here is an apparent factual disagreement bettwen Bayesianism and Frequentism: http://lesswrong.com/lw/ul/my_bayesian_enlightenment/ http://lesswrong.com/lw/ul/my_bayesian_enlightenment/
  - landismj 13y ago
    Third example: A researcher has millions of datasets to analyze, with each dataset containing enough data points such that frequentist asymptotics are satisfied. You are tasked with finding summary statistics for all datasets. The maximum likelihood and maximum a posterior (MAP) estimators are equal within some tolerance for a subset of these datapoints. However, the marginal likelihood function is computationally intractable, so the Bayesian must use expensive methods to produce MAP estimates, e.g. using Markov chain Monte Carlo (MCMC). For complex posterior distributions, MCMC requires careful programming and verification procedures, which can be prohibitive in practice. There are very many real-world problems that have fast and accurate frequentist solutions, but slow and difficult Bayesian solutions. Despite my personal bias -- my research primarily relies on Bayesian inference -- I can't fathom how one can reasonably argue that frequentist approaches are always inferior, even in applied statistics.
capnrefsmmat 13y ago
You may be very interested to read about Bayes factors, which allow interpretable hypothesis tests without prespecified priors: https://www.andrew.cmu.edu/user/kk3n/simplicity/KassRaftery1995.pdf https://www.andrew.cmu.edu/user/kk3n/simplicity/KassRaftery1... (though one does have to specify the hypotheses to be tested more exactly than, say, "the effect is not zero.")

by pja 13y ago

loup-vaillant 13y ago

> And your Bayesian statistician will acquire their priors how exactly? As If Frequentists somehow didn't need priors. Everyone starts with prior knowledge. We might as well use it. Or do you advocate not using every scrap of knowledge available to you? That would be stupid. Sure, prior knowledge can be shaky, or difficult to justify. But at least, a Bayesian will be explicit about it, instead of, like, sweeping normal probability distribution assumptions under the linear regression rug. > it would be a terrible idea to prefer Bayesian approaches to Frequentist ones in all situations. Name three examples that doesn't involve the Frequentist using better prior information than the Bayesian. By the way, Bayesians know that using probability theory correctly is sometimes intractable (combinatorial explosion and all that). In those cases, they will use approximations. But at least, they will know it's an approximation. --- You really should read chapters 1 and 2 of Probability Theory: the Logic of Science. They give a good feel of why Bayesians are correct as a simple matter of fact.

rfergie 13y ago
Those chapters show that any consistent (in some sense) statistical reasoning system must result in the same answers as a Bayesian system. They don't show that the frequentist approach is wrong. If both methods result in the same answer and frequentist methods are easier to use then what is the problem?
- loup-vaillant 13y ago
  No problem of course. But that's a big if. And how are you going to evaluate whether both methods do in fact deliver the same answer? Also, the Frequentist approach is not wrong. It's inaccurate, to the extent its results differ from the Bayesian ones. This inaccuracy tend to go down as we gather more data. Which is a good thing, or else science itself wouldn't work. One more thing. You said "in some sense". Are you seriously suggesting that the assumptions behind Cox's Theorem can reasonably be challenged?
  - thebooktocome 13y ago
    > Are you seriously suggesting that the assumptions behind Cox's Theorem can reasonably be challenged? Sure: probability is continuous.
    - loup-vaillant 13y ago
      You mean, "it is not totally obvious that probability is continuous". Well… To me, it is obvious.
  - rfergie 13y ago
    I'm ashamed to say that I said "in some sense" to be deliberately vague about what I meant by consistency - I didn't want to overstate the strength of Cox's theorem or end up arguing about Godel.
    - loup-vaillant 13y ago
      Actually, I was accusing you of understating the strength of Cox's theorem.
jtleek 13y ago
As the author of the guide in question, I feel I should speak up here. Neither frequentist or Bayesian statistics is "just right". It is 100% dependent on the user and whether they use/interpret the quantities correctly. There are both really good and horrible Frequentist and Bayesian statisticians. To imply one or the other is "better" is incorrect and disingenuous. Just my 2 cents.
- loup-vaillant 13y ago
  Oops. It seems I got carried away. > There are both really good and horrible Frequentist and Bayesian statisticians. Yeah. If I had to choose between Fisher and Anonymous Bayesian, I may chose Fisher. However, unless both kind of statistics yield the same results (I don't think they do), then at least one of them is bogus, by application of the non-contradiction principle. So, while I can imagine there are good Frequentists Statisticians out there, I insist that frequentism itself is bogus.
dspeyer 13y ago
Here's two examples: I'm testing the effectiveness of a drug. Drugs of this class have a certain likelihood of working, the noise in my data is known, the experimental group did this much better than the control... does the drug really work? So far so trivial, in either Bayesianism or Frequentism. Now, I happen to mention that I tested 10000 variants of this drug and only sent data for the one that seemed to work. The rest aren't interesting after all. Under Frequentism, it's easy to take this into account. Under Bayesianism, it requires complex definitions of observations, and is easy to overlook as there's no space for it in the formula. I have a collection of unfair dice. Unfortunately, they all look the same and got dumped on the floor. Now someone grabbed one off the floor at random and wants to make bets with me about it. Even experienced Bayesians are likely to mix up their propositions in a case like this. I say that from having read discussions of similar problems. Yes, if you do it right, it comes out correctly, but Frequentism makes sure you've thought about what you're asking in the same way Bayesianism makes sure you've thought about your priors. Somebody else will have to give a third example. Bayesianism and Frequentism are based on the same math, and math is math. If you use them correctly, they'll get you the same answer every time. The difference is what they make easy, and what mistakes they protect you against.
- loup-vaillant 13y ago
  First example: of course it's complex (though I have no idea what you mean by "definitions of observations"). The correct answer needs my probability distribution over the algorithm your brain used to select which drug to send to me (my brain already hurts). Also, could you describe the Frequentist method in more detail? I'm sure it must overlook something. Second example: Okay, Bayesian statistics are harder. That's a disadvantage. --- > If you use [Bayesianism or Frequentism] correctly, they'll get you the same answer every time Wat. If they gave invariably the same answer, then, why the endless debates? By the way, here is an apparent factual disagreement bettwen Bayesianism and Frequentism: http://lesswrong.com/lw/ul/my_bayesian_enlightenment/ http://lesswrong.com/lw/ul/my_bayesian_enlightenment/
- landismj 13y ago
  Third example: A researcher has millions of datasets to analyze, with each dataset containing enough data points such that frequentist asymptotics are satisfied. You are tasked with finding summary statistics for all datasets. The maximum likelihood and maximum a posterior (MAP) estimators are equal within some tolerance for a subset of these datapoints. However, the marginal likelihood function is computationally intractable, so the Bayesian must use expensive methods to produce MAP estimates, e.g. using Markov chain Monte Carlo (MCMC). For complex posterior distributions, MCMC requires careful programming and verification procedures, which can be prohibitive in practice. There are very many real-world problems that have fast and accurate frequentist solutions, but slow and difficult Bayesian solutions. Despite my personal bias -- my research primarily relies on Bayesian inference -- I can't fathom how one can reasonably argue that frequentist approaches are always inferior, even in applied statistics.

capnrefsmmat 13y ago

You may be very interested to read about Bayes factors, which allow interpretable hypothesis tests without prespecified priors: https://www.andrew.cmu.edu/user/kk3n/simplicity/KassRaftery1995.pdf https://www.andrew.cmu.edu/user/kk3n/simplicity/KassRaftery1... (though one does have to specify the hypotheses to be tested more exactly than, say, "the effect is not zero.")