6 ms·

I don't think this guy understands the debate. A quick summary: If you think statistics is a big toolbox, some of the tools give different answers that are be

by Eliezer 14y ago

I don't think this guy understands the debate. A quick summary: If you think statistics is a big toolbox, some of the tools give different answers that are better or worse in various ways, and you can just take out whatever tool you like, you're a frequentist. If you think that there's such a thing as a correct probability estimate, and all coherent reasoning is required to come up with consistent answers regardless of which different path was taken to arrive at the same destination, you're a Bayesian. From this perspective, a "confidence interval" isn't a tool that's useful on some occasions, it's just plain crazy and wrong, like a weather forecaster who only tells you the probability that it's raining here xor in Narnia. Sure, the forecast is generated by a process that's sorta related to the correct answer, but by manipulating the imaginary land of Narnia you can make the forecast be basically anything. With Bayesianism there are no degrees of freedom in the likelihood ratio you report. See http://xkcd.com/1132/ http://xkcd.com/1132/. It doesn't do any good to appeal to the idea that Bayesian methods are just one tool in the toolbox. Only frequentists think in terms of toolboxes in the first place. Also Bayes's Rule is tautologically equivalent to Bayes's Theorem. There's more wrong, but meanwhile, color me unimpressed.

keithwinstein 14y ago
I wouldn't want to go up against Eliezer Yudkowsky casually, but here goes: the guy is basically correct. (Although I also didn't follow his statement that Bayes Theorem != Bayes Rule.) You can see my essay here, where I express a similar view: http://www.quora.com/What-is-the-difference-between-Bayesian-and-frequentist-statisticians http://www.quora.com/What-is-the-difference-between-Bayesian... Confidence intervals and credibility intervals are both mathematical objects that have well-specified (and different) properties. Confidence intervals are a worst-case technique and posterior probabilities are a sort of average-case technique. It's not "wrong" to say that the worst-case runtime of QuickSort is O(n^2) and it's not wrong to say that, given a uniform probability distribution over inputs, the expected runtime is O(n log n). Which statement is useful to make depends on your requirements. They're both true. In my "100 independent robots" example, for instance, the credibility interval or posterior probability produces answers that are not helpful for the application (and not particularly intuitive either -- the 70% credibility interval is "wrong" 80% of the time, given a certain value of the parameter). This can be a question of engineering and there's no need to be dogmatic about it. It is perverse and incorrect to say that "Only frequentists think in terms of toolboxes"! All mathematicians and engineers have access to the whole world of theorems and algorithms and techniques, all of them true, or meeting their specifications. No mathematician would argue that the Chinese remainder theorem is wrong because they are a Galois theorist! And no practitioner of Bayesian methods should argue that confidence intervals that meet their worst-case coverage guarantee are "wrong" because the person uses posterior probabilities. (My quibble here is with dogmatism, not with Bayesians, because the dogmatic frequentists are just as bad. They just don't hang out on Hacker News.)
- tel 14y ago
  I agree with you and Larry and want to elaborate. Dogmatic Bayesians often come armed with assertions that probability calculus is the "correct" way of doing inference and, further, that there is some kind of ethical maximum achieved when you make your decisions exclusively via inference. They have strong reason to feel this way, of course. Probability calculus is a very elegant theory that can be constructed from rather sane "desiderata" a la Cox's Theorem. Furthermore, if you think of ethics as an optimization problem, which is fair, then you're likely to be attracted to decision theory as a powerful tool for being more ethical. QED? Unfortunately, neither of these reasons have the conclusive force that they appear to. The sanity of Cox's postulates and ethics of optimization are still (arbitrary) choices which serve mostly to create a formalized world for us to live in and study with efficacy. There is no doubt that people have derived great power from similar use of powerful models, but models they are nonetheless. Asserting that modeling the world based on these postulates is ethically mandatory is weird. Feeling confident that you can become more powerful than others by doing so is a bet. So, assuming you're a betting type instead of a religious type, would you rather use models that are elegant but incomplete or practical and battle-tested? I believe both Sampling philosophy and Bayesian philosophy to be mathematically elegant. I also have seen them both very practically used. And once you're interested in measuring practical performance of these methods, once you decide to pick things like loss functions and philosophies of measurement, then it's easy to find places where either framework fails. --- So really, in my mind, there are Bayesian methods and Frequentist methods and Bayesians who, probably following Jaynes' hyperbolic and recommended work, feel that there is a certain theory of mind which is correct due to its elegance. Most Engineers can pick between the methods and most mathematicians can pick between the theories. And having a favorite theory isn't so bad either. I quite like some "Likelihoodists" as well and learn from them every time we talk. --- I think the best model of the situation is that of logic. It's like we have "First-order logicians" and "Higher-order logicians", each of which espousing a viewpoint that a particular Theory is "more true" than the other. Truly, they both have fortes and foibles, though, and the practical mathematician studies both and uses whichever one lets them prove what they need to. The problem comes into play when "higher-order logicians" suggest that "first-order logicians" are inherently wrong since their very theory prevents them from being able to conceive of a better theory and this reticence is leading to disarray. But then the "higher-order logicians" can't come to a consensus on how to fix things and also take a potentially infinite amount of time to reach a conclusion. Really, though, our thought is neither bound to first or second order logic. Perhaps this causes us pain from time to time, but it also lets us pick and choose.

by Eliezer 14y ago

keithwinstein 14y ago

I wouldn't want to go up against Eliezer Yudkowsky casually, but here goes: the guy is basically correct. (Although I also didn't follow his statement that Bayes Theorem != Bayes Rule.) You can see my essay here, where I express a similar view: http://www.quora.com/What-is-the-difference-between-Bayesian-and-frequentist-statisticians http://www.quora.com/What-is-the-difference-between-Bayesian... Confidence intervals and credibility intervals are both mathematical objects that have well-specified (and different) properties. Confidence intervals are a worst-case technique and posterior probabilities are a sort of average-case technique. It's not "wrong" to say that the worst-case runtime of QuickSort is O(n^2) and it's not wrong to say that, given a uniform probability distribution over inputs, the expected runtime is O(n log n). Which statement is useful to make depends on your requirements. They're both true. In my "100 independent robots" example, for instance, the credibility interval or posterior probability produces answers that are not helpful for the application (and not particularly intuitive either -- the 70% credibility interval is "wrong" 80% of the time, given a certain value of the parameter). This can be a question of engineering and there's no need to be dogmatic about it. It is perverse and incorrect to say that "Only frequentists think in terms of toolboxes"! All mathematicians and engineers have access to the whole world of theorems and algorithms and techniques, all of them true, or meeting their specifications. No mathematician would argue that the Chinese remainder theorem is wrong because they are a Galois theorist! And no practitioner of Bayesian methods should argue that confidence intervals that meet their worst-case coverage guarantee are "wrong" because the person uses posterior probabilities. (My quibble here is with dogmatism, not with Bayesians, because the dogmatic frequentists are just as bad. They just don't hang out on Hacker News.)

tel 14y ago
I agree with you and Larry and want to elaborate. Dogmatic Bayesians often come armed with assertions that probability calculus is the "correct" way of doing inference and, further, that there is some kind of ethical maximum achieved when you make your decisions exclusively via inference. They have strong reason to feel this way, of course. Probability calculus is a very elegant theory that can be constructed from rather sane "desiderata" a la Cox's Theorem. Furthermore, if you think of ethics as an optimization problem, which is fair, then you're likely to be attracted to decision theory as a powerful tool for being more ethical. QED? Unfortunately, neither of these reasons have the conclusive force that they appear to. The sanity of Cox's postulates and ethics of optimization are still (arbitrary) choices which serve mostly to create a formalized world for us to live in and study with efficacy. There is no doubt that people have derived great power from similar use of powerful models, but models they are nonetheless. Asserting that modeling the world based on these postulates is ethically mandatory is weird. Feeling confident that you can become more powerful than others by doing so is a bet. So, assuming you're a betting type instead of a religious type, would you rather use models that are elegant but incomplete or practical and battle-tested? I believe both Sampling philosophy and Bayesian philosophy to be mathematically elegant. I also have seen them both very practically used. And once you're interested in measuring practical performance of these methods, once you decide to pick things like loss functions and philosophies of measurement, then it's easy to find places where either framework fails. --- So really, in my mind, there are Bayesian methods and Frequentist methods and Bayesians who, probably following Jaynes' hyperbolic and recommended work, feel that there is a certain theory of mind which is correct due to its elegance. Most Engineers can pick between the methods and most mathematicians can pick between the theories. And having a favorite theory isn't so bad either. I quite like some "Likelihoodists" as well and learn from them every time we talk. --- I think the best model of the situation is that of logic. It's like we have "First-order logicians" and "Higher-order logicians", each of which espousing a viewpoint that a particular Theory is "more true" than the other. Truly, they both have fortes and foibles, though, and the practical mathematician studies both and uses whichever one lets them prove what they need to. The problem comes into play when "higher-order logicians" suggest that "first-order logicians" are inherently wrong since their very theory prevents them from being able to conceive of a better theory and this reticence is leading to disarray. But then the "higher-order logicians" can't come to a consensus on how to fix things and also take a potentially infinite amount of time to reach a conclusion. Really, though, our thought is neither bound to first or second order logic. Perhaps this causes us pain from time to time, but it also lets us pick and choose.