4 ms·

Here's two examples: I'm testing the effectiveness of a drug. Drugs of this class have a certain likelihood of working, the noise in my data is known, the exp

by dspeyer 13y ago

Here's two examples: I'm testing the effectiveness of a drug. Drugs of this class have a certain likelihood of working, the noise in my data is known, the experimental group did this much better than the control... does the drug really work? So far so trivial, in either Bayesianism or Frequentism. Now, I happen to mention that I tested 10000 variants of this drug and only sent data for the one that seemed to work. The rest aren't interesting after all. Under Frequentism, it's easy to take this into account. Under Bayesianism, it requires complex definitions of observations, and is easy to overlook as there's no space for it in the formula. I have a collection of unfair dice. Unfortunately, they all look the same and got dumped on the floor. Now someone grabbed one off the floor at random and wants to make bets with me about it. Even experienced Bayesians are likely to mix up their propositions in a case like this. I say that from having read discussions of similar problems. Yes, if you do it right, it comes out correctly, but Frequentism makes sure you've thought about what you're asking in the same way Bayesianism makes sure you've thought about your priors. Somebody else will have to give a third example. Bayesianism and Frequentism are based on the same math, and math is math. If you use them correctly, they'll get you the same answer every time. The difference is what they make easy, and what mistakes they protect you against.

loup-vaillant 13y ago
First example: of course it's complex (though I have no idea what you mean by "definitions of observations"). The correct answer needs my probability distribution over the algorithm your brain used to select which drug to send to me (my brain already hurts). Also, could you describe the Frequentist method in more detail? I'm sure it must overlook something. Second example: Okay, Bayesian statistics are harder. That's a disadvantage. --- > If you use [Bayesianism or Frequentism] correctly, they'll get you the same answer every time Wat. If they gave invariably the same answer, then, why the endless debates? By the way, here is an apparent factual disagreement bettwen Bayesianism and Frequentism: http://lesswrong.com/lw/ul/my_bayesian_enlightenment/ http://lesswrong.com/lw/ul/my_bayesian_enlightenment/
landismj 13y ago
Third example: A researcher has millions of datasets to analyze, with each dataset containing enough data points such that frequentist asymptotics are satisfied. You are tasked with finding summary statistics for all datasets. The maximum likelihood and maximum a posterior (MAP) estimators are equal within some tolerance for a subset of these datapoints. However, the marginal likelihood function is computationally intractable, so the Bayesian must use expensive methods to produce MAP estimates, e.g. using Markov chain Monte Carlo (MCMC). For complex posterior distributions, MCMC requires careful programming and verification procedures, which can be prohibitive in practice. There are very many real-world problems that have fast and accurate frequentist solutions, but slow and difficult Bayesian solutions. Despite my personal bias -- my research primarily relies on Bayesian inference -- I can't fathom how one can reasonably argue that frequentist approaches are always inferior, even in applied statistics.
- loup-vaillant 13y ago
  > I can't fathom how one can reasonably argue that frequentist approaches are always inferior, even in applied statistics. My original claim is broader than I wanted it to be. The fact is, a Frequentist approach will always be less accurate than the correct application of probability theory. But of course, > Bayesians know that using probability theory correctly is sometimes intractable (combinatorial explosion and all that). In those cases, they will use approximations. But at least, they will know it's an approximation. https://news.ycombinator.com/item?id=6793905 https://news.ycombinator.com/item?id=6793905 The key to the Bayesian outlook is to remember that no matter what, there is a correct answer, even if you can't afford to compute it. As Eliezer Yudkowsky put it, there are laws of thought. Want to use Frequentist tools? Sure, why not. Just remember that they often violate the laws of ideal though. Some inaccuracy inevitably ensues.

by dspeyer 13y ago

loup-vaillant 13y ago

First example: of course it's complex (though I have no idea what you mean by "definitions of observations"). The correct answer needs my probability distribution over the algorithm your brain used to select which drug to send to me (my brain already hurts). Also, could you describe the Frequentist method in more detail? I'm sure it must overlook something. Second example: Okay, Bayesian statistics are harder. That's a disadvantage. --- > If you use [Bayesianism or Frequentism] correctly, they'll get you the same answer every time Wat. If they gave invariably the same answer, then, why the endless debates? By the way, here is an apparent factual disagreement bettwen Bayesianism and Frequentism: http://lesswrong.com/lw/ul/my_bayesian_enlightenment/ http://lesswrong.com/lw/ul/my_bayesian_enlightenment/

landismj 13y ago

Third example: A researcher has millions of datasets to analyze, with each dataset containing enough data points such that frequentist asymptotics are satisfied. You are tasked with finding summary statistics for all datasets. The maximum likelihood and maximum a posterior (MAP) estimators are equal within some tolerance for a subset of these datapoints. However, the marginal likelihood function is computationally intractable, so the Bayesian must use expensive methods to produce MAP estimates, e.g. using Markov chain Monte Carlo (MCMC). For complex posterior distributions, MCMC requires careful programming and verification procedures, which can be prohibitive in practice. There are very many real-world problems that have fast and accurate frequentist solutions, but slow and difficult Bayesian solutions. Despite my personal bias -- my research primarily relies on Bayesian inference -- I can't fathom how one can reasonably argue that frequentist approaches are always inferior, even in applied statistics.

loup-vaillant 13y ago
> I can't fathom how one can reasonably argue that frequentist approaches are always inferior, even in applied statistics. My original claim is broader than I wanted it to be. The fact is, a Frequentist approach will always be less accurate than the correct application of probability theory. But of course, > Bayesians know that using probability theory correctly is sometimes intractable (combinatorial explosion and all that). In those cases, they will use approximations. But at least, they will know it's an approximation. https://news.ycombinator.com/item?id=6793905 https://news.ycombinator.com/item?id=6793905 The key to the Bayesian outlook is to remember that no matter what, there is a correct answer, even if you can't afford to compute it. As Eliezer Yudkowsky put it, there are laws of thought. Want to use Frequentist tools? Sure, why not. Just remember that they often violate the laws of ideal though. Some inaccuracy inevitably ensues.