6 ms·

Bayesian vs. frequentist: squabbling among the ignorant

110 points by madhadron 12y ago

bjterry 12y ago
This article stands in contrast to the thoughts expressed in a recent interview with George Ellis[1]. The question of Bayesian vs. frequentist is fundamentally a philosophical question, and it's often discussed in these terms in the articles about the debate, including the recent one which caused this post. The philosophy that we accept affects the tools that we use to solve problems, and the way we think about problems, so you can't simply ignore the debate. Ellis' argument even extends to his example, about ψ (the quantum wave function), about which there is extensive philosophical debate. When people identify as supporting Bayesian methods over frequentist methods, that actually changes the way they perform science. It's an argument against bad statistical interpretations that frequently arise in the social sciences and medicine[2] and elsewhere because p-values have become the altar at which all publishing researchers must prostrate themselves. It's not a case of a debate without substance, which would have no observable effects. I can understand the frustrations of the author. Every tool has its use, and frequentist statistical methods are frequently useful. They help you get published, they are broadly supported by software packages, they are often faster and easier to perform, etc., so there are many situations in which they are the right choice. But I feel like in attempting to be agnostic about methods, the author is losing nuance in their argument. Bayesian reasoning really does represent the ground truth (assuming imperfect knowledge), and even when we are using frequentist methods, we are only doing so because the benefits outweigh the costs. 1: http://blogs.scientificamerican.com/cross-check/2014/08/21/quantum-gravity-expert-says-philosophical-superficiality-has-harmed-physics/?-k http://blogs.scientificamerican.com/cross-check/2014/08/21/q... 2: http://www.sciencebasedmedicine.org/prior-probability-the-dirty-little-secret-of-evidence-based-alternative-medicine-2/ http://www.sciencebasedmedicine.org/prior-probability-the-di...
- keithwinstein 12y ago
  Don't think that's quite what the author was getting at. A "frequentist" who makes a decision based on a p-value or confidence interval, and a "Bayesian" who makes a decision based on a posterior or predictive probability distribution, can both be viewed as making a decision according to a procedure that minimizes some statistic of a loss function. Both of these broad families of methods are diverse, but to generalize broadly, decisions informed by "frequentist" tools are often concerned with the worst-case value of the loss function within some universe, and decisions based on "Bayesian" methods can be viewed as caring about the expected value of the loss function given some prior. Both of those families (and many others) are "the ground truth," in that they both make statements that can be proved as theorems of mathematics. A "frequentist" confidence interval will always achieve its guaranteed coverage no matter what, if the likelihood function is true. A "Bayesian" credibility interval will include the true value of the parameters at exactly the advertised rate, when averaged over each possible value of the parameters weighted according to the prior, and assuming the likelihood function is true. The author says, and I agree, that the thing worth arguing over is the "norm we use to choose our optimal procedure" and the nature of the loss function. Whether you care about controlling the worst case or the expected value or something else depends on what matters. (The author points out that in an adversarial situation where your strategy is known to your opponent, minimizing the worst case may be advisable...) Some days, we care about the worst-case performance of QuickSort, and some days we care about its average-case performance (given an assumption that all input orderings are uniformly likely). It's ok to care about different things depending on the application; we don't have to split into warring tribes over it. More here: http://blog.keithw.org/2013/02/q-what-is-difference-between-bayesian.html http://blog.keithw.org/2013/02/q-what-is-difference-between-... http://jsteinhardt.wordpress.com/2014/02/10/a-fervent-defense-of-frequentist-statistics/ http://jsteinhardt.wordpress.com/2014/02/10/a-fervent-defens... http://cs.stanford.edu/~jsteinhardt/stats-essay.pdf http://cs.stanford.edu/~jsteinhardt/stats-essay.pdf
  - Matumio 12y ago
    Do Bayesian methods imply a choice of loss function? I think it's a framework for observational statistics, not for policy making or decision theory. You can use it as a tool during policy or decision making, where you should indeed have a debate about picking a loss function that corresponds to your goals.
    - mjw 12y ago
      If you're reporting point or interval estimates (rather than the entire posterior) then you are implicitly or explicitly optimising some kind of loss function. Also, worth a reminder that continuous estimation problems can be viewed as decision problems too, it's not just about discrete decision-making. I don't think it's great to take the view that, because I'm not making a decision based on this estimate myself, I don't need to worry about which loss function I'm implicitly optimising for when choosing an estimator. Someone else may need to.
  - jules 12y ago
    Bayesian statistics gives you a posterior distribution. What you do with that distribution is up to you. If you want to find the decision that minimizes the maximum loss instead of the expected loss, that fits perfectly well within the Bayesian framework. The posterior gives you all the information that you need to make a decision, whatever your loss function. Frequentist statistics on the other hand gives you no such thing. You can't make decisions based on frequentist statistics. Frequentist statistics is all about reasoning about things that did not happen. Things that did not happen are irrelevant for making decisions in a situation where that thing did happen. So while frequentist statistics is mathematically correct, it's also strictly speaking useless in practice. It's only useful insofar as it gives us heuristics for decision making when the Bayesian approach is impractical. Here's an example. Suppose there are two types of berries: edible and poisonous. We devise a statistical procedure where you measure some properties of the berry (lets say we look at the color), and the procedure should help you decide whether to eat that berry. Now in frequentist statistics, you'll get a procedure that gives you the correct answer with probability at least p regardless of what your measurement was. Suppose that p=90% and we observe that the color is blue, and the procedure says: this berry is edible. Can we now eat the berry? No! This says absolutely nothing about the probability of the blue berry being poisonous or not. For example suppose 95% of berries are edible and red, and 5% of the berries are poisonous and blue. Then a valid procedure would be one that says that all berries are edible. It's correct >90% of the time, so yay! But if the berry we are holding in our hands is blue, it's incorrect 100% of the time. The fact that the procedure would have given us the right answer if the berry we found was red is irrelevant for making the decision of whether to eat the berry in the situation where the berry we found was blue. Things that did not happen are irrelevant for making decisions in a situation where that thing did happen! tl;dr: frequentist vs bayesian is not about worst case vs average case. It's about P(measurement | true value) vs P(true value | measurement). The former is irrelevant for making decisions, the latter is exactly what you want.
- scotty79 12y ago
  If you are restricting your tools on the grounds of philosophy you subscribe to, you are doing it wrong. Ditch the philosophy, stick to math, try everything and go with what results in models of predictive utility. That's what actual scientists do.
  - analog31 12y ago
    ... and test the prediction.
    - TeMPOraL 12y ago
      That is implied in model being of predictive utility.
  - judk 12y ago
    How do you decide which models actually have predictive utility, under real world incomplete information? Back to square One.
    - TeMPOraL 12y ago
      How do you decide if the water in your bathtub is warm? Stick your finger into it and see for yourself. Similarly here, you use your models to predict something, check if they predicted it right, and either try to fix them or ditch them completely.
  - eruditely 12y ago
    Incorrect. The philosophers know 'math'. The half-mathematicians that think they can escape philosophy should read.... well anything. Almost all grand-standing that people have about 'philosophy' is some weird fictional idea that people have in their mind. Notice how all the recipients of the Lakatos awards and how they are people like Judea Pearl. Here's a quote from here (Terrence Tao, definitely a mathematician). http://terrytao.wordpress.com/2010/01/01/254a-notes-0-a-review-of-probability-theory/ http://terrytao.wordpress.com/2010/01/01/254a-notes-0-a-revi... "probability theory is only “allowed” to study concepts and perform operations which are preserved with respect to extension of the underlying sample space." Any attempt to side-step these issues leads to people wandering off into the middle of nowhere. Misuse of mathematical tools to areas where is does not apply and all sorts of other major and minor errors that could have been avoided. There is no computer science without Godel, Frege, Russell, and Turing.
- wirrbel 12y ago
  As a bayesianist I would like to defend Frequentism here. p-value hacking IS NOT "Frequentist", it is malpractice has its roots in lack of education (copy-paste-analyses), publication pressure and using the p-value as a strong metric for judgement of results (which it was never intended to be). Frequentism is a toolset of best practices and experiences about statistics, a KISS approach to statistics so to speak. Bayesianism is a beautiful approach to probabilities and statistics, yet often turns out to be really complex when performing calculations and it is also not fool-proof and it also requires experience to be applied correctly.
jmpeax 12y ago
...and yet another ignoramus who doesn't understand the debate and so blankets it with the predictably safe "you're both wrong". Almost stopped reading at the hilariously rubbish statement "All the models have limitations which make them of useless in practice." but it's Sunday, and I'm being entertained.
- andrea_s 12y ago
  I think "of" meant to stand for "often". Which is still a very strong position, but not entirely out of the park - it's quite common (not only in statistics, of course) to encounter situations where not every prerequisite of a certain methodology is met, and yet the obtained results are usable in a practical environment.
papaf 12y ago
The book mentioned in the discussion looks interesting but super expensive: http://www.amazon.com/Introduction-Statistical-Inference-Springer-Statistics/dp/1461395801 http://www.amazon.com/Introduction-Statistical-Inference-Spr... Has anyone read it?
- raverbashing 12y ago
  Try your nearest university classifieds or used book store
- RoboTeddy 12y ago
  looks like you can find it on http://booksprice.com http://booksprice.com for about $50
eemax 12y ago
Are there really people other than philosophers who are actually "Bayesians" or "Frequentists"? Frequentist statistics are just a special case of Bayesian statistics with certain implicit, built-in priors - if you pick these priors using Bayesian statistics you'll get the same answers. The frequentest toolbox is just a collection of these useful special cases of Bayesian statistics with priors that usually make sense in practice. The advantage is this greatly simplifies the statistical analysis for many problems. Sometimes these methods fail though, when the priors they rely on implicitly are far off from the actual, and so a Bayesian analysis is needed. Of course, sometimes it's difficult or impossible to use Bayesian methods. It's analogous to classical vs. quantum/relativistic physics. For many cases, you can get the right answer to a problem using classical physics. But under certain conditions, classical physics breaks down, and you must apply quantum or relativistic physics to get a meaningful answer. On the other hand, for many problems it would be silly or impossible to use quantum or relativistic physics because classical is perfectly good. So you might get into argument about whether a specific case can be adequately handled by frequentest statistical methods, or whether a bayesian analysis can/must be applied. The philosophical debate about the different approaches to the nature of probability is just that - a philosophical one, and one that has no real bearing on the usefulness and correctness of bayesian or frequentist statistics in practice.
- keithwinstein 12y ago
  I'm afraid you're mistaken -- frequentist statistics are not a special case of Bayesian statistics, and there is generally no "implicit, built-in prior" that allows you to get frequentist worst-case guarantees from Bayesian techniques. Consider the case of a confidence interval on a binary proportion given a finite number of samples. A frequentist method will produce an interval that includes the true value of the proportion with at least x% probability, even in the worst case, for any proportion between 0 and 1. (E.g. the Blyth-Still-Casella method or the Clopper-Pearson method.) An x% credible interval will include the true value exactly x% of the time, averaged over all values of the proportion weighted according to the prior. This will not provide the same guarantee (much less the same interval extent!), no matter what prior is used. Which is not to say the credible interval is bad or inappropriate. It's just not the same kind of tool. It is optimizing a different penalty function of non-inclusion. (Another example where the "Bayesian" technique is not quite as conservative as necessary to achieve a frequentist-style guarantee, even with a uniform prior: http://www.quora.com/I-have-burned-200-disks-and-I-want-to-make-sure-that-they-are-all-in-perfect-working-order-What-is-the-smallest-size-sample-I-could-test-in-order-to-be-relatively-confident-that-98-of-all-the-disks-are-fine-burned-correctly?share=1 http://www.quora.com/I-have-burned-200-disks-and-I-want-to-m...)
song 12y ago
I always think that having a better understanding of statistics and the mathematics associated with Bayesian calculations would be useful but I don't know where to start. What would be a good book for me to learn?
- Matumio 12y ago
  Depends on your background. For me Chapter 28 about "Model Comparison and Occam's Razor" in MacKay's book was an eye-opener regarding the usefulness of Bayesian methods. http://www.inference.phy.cam.ac.uk/itprnn/book.html http://www.inference.phy.cam.ac.uk/itprnn/book.html This book is good if you are interested in information theory and machine learning, but it assumes a basic math and probability background.
- tmoertel 12y ago
  Probability Theory: The Logic of Science, by E.T. Jaynes, is a highly readable definitive reference that builds the theory from the ground up. The first three chapters are all you need for your purposes and are available online for free (link below), but the entire book is wonderful and well worth reading if you are serious about the subject. http://bayes.wustl.edu/etj/prob/book.pdf http://bayes.wustl.edu/etj/prob/book.pdf
bayesianhorse 12y ago
So if two camps of nerds argue, inevitably someone has to submit a post to hackernews "You are both wrong! Let me show you why ..." And maybe this post also rubs me a little wrong because on the one hand I have an interest in bayesian methods and on the other I will never be able to get a thorough nit-picking-able math education...
judk 12y ago
At the end of the day, statistics is about using a rationally consistent process to making decisions under uncertainty, but everyone (except the theoreticians) tries to use it to make correct decisions, conjuring certainty from the ether, and then railing against anyone who disagrees with their choice of articles of faith. In applied statistics we find the bitter religious rivalries of science.
Double_Cast 12y ago
> You prove your understanding by the type of questions you ask. I can't find where I paraphrased this from. Maybe it was Better Explained [0] while explaining the intuition behind complex numbers? As I remember, the context was "Now you understand that complex numbers are rotations through 2 dimensions, I bet a lot of you are asking 'can we extend math to rotations through 3 dimensions?'" (aka quaternions) The vibe I get from this article is "the meaning isn't important, so plug and chug away". No no no no no! Grokking the meaning is important because it allows us to extend our understanding from a strong foundation. E.g. HN a few days ago submitted a paper on "Half Coins"[1] which extends our conventional notion of probability to negative numbers. And after I read the paper, I realized it's not as weird as it sounds. [0] http://betterexplained.com/cheatsheet/ http://betterexplained.com/cheatsheet/ [1] https://news.ycombinator.com/item?id=8187457 https://news.ycombinator.com/item?id=8187457 disclaimer: I don't know anything about statistics.

110 points by madhadron 12y ago

This article stands in contrast to the thoughts expressed in a recent interview with George Ellis[1]. The question of Bayesian vs. frequentist is fundamentally a philosophical question, and it's often discussed in these terms in the articles about the debate, including the recent one which caused this post. The philosophy that we accept affects the tools that we use to solve problems, and the way we think about problems, so you can't simply ignore the debate. Ellis' argument even extends to his example, about ψ (the quantum wave function), about which there is extensive philosophical debate. When people identify as supporting Bayesian methods over frequentist methods, that actually changes the way they perform science. It's an argument against bad statistical interpretations that frequently arise in the social sciences and medicine[2] and elsewhere because p-values have become the altar at which all publishing researchers must prostrate themselves. It's not a case of a debate without substance, which would have no observable effects. I can understand the frustrations of the author. Every tool has its use, and frequentist statistical methods are frequently useful. They help you get published, they are broadly supported by software packages, they are often faster and easier to perform, etc., so there are many situations in which they are the right choice. But I feel like in attempting to be agnostic about methods, the author is losing nuance in their argument. Bayesian reasoning really does represent the ground truth (assuming imperfect knowledge), and even when we are using frequentist methods, we are only doing so because the benefits outweigh the costs. 1: http://blogs.scientificamerican.com/cross-check/2014/08/21/quantum-gravity-expert-says-philosophical-superficiality-has-harmed-physics/?-k http://blogs.scientificamerican.com/cross-check/2014/08/21/q... 2: http://www.sciencebasedmedicine.org/prior-probability-the-dirty-little-secret-of-evidence-based-alternative-medicine-2/ http://www.sciencebasedmedicine.org/prior-probability-the-di...

keithwinstein 12y ago
Don't think that's quite what the author was getting at. A "frequentist" who makes a decision based on a p-value or confidence interval, and a "Bayesian" who makes a decision based on a posterior or predictive probability distribution, can both be viewed as making a decision according to a procedure that minimizes some statistic of a loss function. Both of these broad families of methods are diverse, but to generalize broadly, decisions informed by "frequentist" tools are often concerned with the worst-case value of the loss function within some universe, and decisions based on "Bayesian" methods can be viewed as caring about the expected value of the loss function given some prior. Both of those families (and many others) are "the ground truth," in that they both make statements that can be proved as theorems of mathematics. A "frequentist" confidence interval will always achieve its guaranteed coverage no matter what, if the likelihood function is true. A "Bayesian" credibility interval will include the true value of the parameters at exactly the advertised rate, when averaged over each possible value of the parameters weighted according to the prior, and assuming the likelihood function is true. The author says, and I agree, that the thing worth arguing over is the "norm we use to choose our optimal procedure" and the nature of the loss function. Whether you care about controlling the worst case or the expected value or something else depends on what matters. (The author points out that in an adversarial situation where your strategy is known to your opponent, minimizing the worst case may be advisable...) Some days, we care about the worst-case performance of QuickSort, and some days we care about its average-case performance (given an assumption that all input orderings are uniformly likely). It's ok to care about different things depending on the application; we don't have to split into warring tribes over it. More here: http://blog.keithw.org/2013/02/q-what-is-difference-between-bayesian.html http://blog.keithw.org/2013/02/q-what-is-difference-between-... http://jsteinhardt.wordpress.com/2014/02/10/a-fervent-defense-of-frequentist-statistics/ http://jsteinhardt.wordpress.com/2014/02/10/a-fervent-defens... http://cs.stanford.edu/~jsteinhardt/stats-essay.pdf http://cs.stanford.edu/~jsteinhardt/stats-essay.pdf
- Matumio 12y ago
  Do Bayesian methods imply a choice of loss function? I think it's a framework for observational statistics, not for policy making or decision theory. You can use it as a tool during policy or decision making, where you should indeed have a debate about picking a loss function that corresponds to your goals.
  - mjw 12y ago
    If you're reporting point or interval estimates (rather than the entire posterior) then you are implicitly or explicitly optimising some kind of loss function. Also, worth a reminder that continuous estimation problems can be viewed as decision problems too, it's not just about discrete decision-making. I don't think it's great to take the view that, because I'm not making a decision based on this estimate myself, I don't need to worry about which loss function I'm implicitly optimising for when choosing an estimator. Someone else may need to.
- jules 12y ago
  Bayesian statistics gives you a posterior distribution. What you do with that distribution is up to you. If you want to find the decision that minimizes the maximum loss instead of the expected loss, that fits perfectly well within the Bayesian framework. The posterior gives you all the information that you need to make a decision, whatever your loss function. Frequentist statistics on the other hand gives you no such thing. You can't make decisions based on frequentist statistics. Frequentist statistics is all about reasoning about things that did not happen. Things that did not happen are irrelevant for making decisions in a situation where that thing did happen. So while frequentist statistics is mathematically correct, it's also strictly speaking useless in practice. It's only useful insofar as it gives us heuristics for decision making when the Bayesian approach is impractical. Here's an example. Suppose there are two types of berries: edible and poisonous. We devise a statistical procedure where you measure some properties of the berry (lets say we look at the color), and the procedure should help you decide whether to eat that berry. Now in frequentist statistics, you'll get a procedure that gives you the correct answer with probability at least p regardless of what your measurement was. Suppose that p=90% and we observe that the color is blue, and the procedure says: this berry is edible. Can we now eat the berry? No! This says absolutely nothing about the probability of the blue berry being poisonous or not. For example suppose 95% of berries are edible and red, and 5% of the berries are poisonous and blue. Then a valid procedure would be one that says that all berries are edible. It's correct >90% of the time, so yay! But if the berry we are holding in our hands is blue, it's incorrect 100% of the time. The fact that the procedure would have given us the right answer if the berry we found was red is irrelevant for making the decision of whether to eat the berry in the situation where the berry we found was blue. Things that did not happen are irrelevant for making decisions in a situation where that thing did happen! tl;dr: frequentist vs bayesian is not about worst case vs average case. It's about P(measurement | true value) vs P(true value | measurement). The former is irrelevant for making decisions, the latter is exactly what you want.
scotty79 12y ago
If you are restricting your tools on the grounds of philosophy you subscribe to, you are doing it wrong. Ditch the philosophy, stick to math, try everything and go with what results in models of predictive utility. That's what actual scientists do.
- analog31 12y ago
  ... and test the prediction.
  - TeMPOraL 12y ago
    That is implied in model being of predictive utility.
- judk 12y ago
  How do you decide which models actually have predictive utility, under real world incomplete information? Back to square One.
  - TeMPOraL 12y ago
    How do you decide if the water in your bathtub is warm? Stick your finger into it and see for yourself. Similarly here, you use your models to predict something, check if they predicted it right, and either try to fix them or ditch them completely.
- eruditely 12y ago
  Incorrect. The philosophers know 'math'. The half-mathematicians that think they can escape philosophy should read.... well anything. Almost all grand-standing that people have about 'philosophy' is some weird fictional idea that people have in their mind. Notice how all the recipients of the Lakatos awards and how they are people like Judea Pearl. Here's a quote from here (Terrence Tao, definitely a mathematician). http://terrytao.wordpress.com/2010/01/01/254a-notes-0-a-review-of-probability-theory/ http://terrytao.wordpress.com/2010/01/01/254a-notes-0-a-revi... "probability theory is only “allowed” to study concepts and perform operations which are preserved with respect to extension of the underlying sample space." Any attempt to side-step these issues leads to people wandering off into the middle of nowhere. Misuse of mathematical tools to areas where is does not apply and all sorts of other major and minor errors that could have been avoided. There is no computer science without Godel, Frege, Russell, and Turing.
wirrbel 12y ago
As a bayesianist I would like to defend Frequentism here. p-value hacking IS NOT "Frequentist", it is malpractice has its roots in lack of education (copy-paste-analyses), publication pressure and using the p-value as a strong metric for judgement of results (which it was never intended to be). Frequentism is a toolset of best practices and experiences about statistics, a KISS approach to statistics so to speak. Bayesianism is a beautiful approach to probabilities and statistics, yet often turns out to be really complex when performing calculations and it is also not fool-proof and it also requires experience to be applied correctly.

jmpeax 12y ago

...and yet another ignoramus who doesn't understand the debate and so blankets it with the predictably safe "you're both wrong". Almost stopped reading at the hilariously rubbish statement "All the models have limitations which make them of useless in practice." but it's Sunday, and I'm being entertained.

andrea_s 12y ago
I think "of" meant to stand for "often". Which is still a very strong position, but not entirely out of the park - it's quite common (not only in statistics, of course) to encounter situations where not every prerequisite of a certain methodology is met, and yet the obtained results are usable in a practical environment.

papaf 12y ago

The book mentioned in the discussion looks interesting but super expensive: http://www.amazon.com/Introduction-Statistical-Inference-Springer-Statistics/dp/1461395801 http://www.amazon.com/Introduction-Statistical-Inference-Spr... Has anyone read it?

raverbashing 12y ago
Try your nearest university classifieds or used book store
RoboTeddy 12y ago
looks like you can find it on http://booksprice.com http://booksprice.com for about $50

eemax 12y ago

Are there really people other than philosophers who are actually "Bayesians" or "Frequentists"? Frequentist statistics are just a special case of Bayesian statistics with certain implicit, built-in priors - if you pick these priors using Bayesian statistics you'll get the same answers. The frequentest toolbox is just a collection of these useful special cases of Bayesian statistics with priors that usually make sense in practice. The advantage is this greatly simplifies the statistical analysis for many problems. Sometimes these methods fail though, when the priors they rely on implicitly are far off from the actual, and so a Bayesian analysis is needed. Of course, sometimes it's difficult or impossible to use Bayesian methods. It's analogous to classical vs. quantum/relativistic physics. For many cases, you can get the right answer to a problem using classical physics. But under certain conditions, classical physics breaks down, and you must apply quantum or relativistic physics to get a meaningful answer. On the other hand, for many problems it would be silly or impossible to use quantum or relativistic physics because classical is perfectly good. So you might get into argument about whether a specific case can be adequately handled by frequentest statistical methods, or whether a bayesian analysis can/must be applied. The philosophical debate about the different approaches to the nature of probability is just that - a philosophical one, and one that has no real bearing on the usefulness and correctness of bayesian or frequentist statistics in practice.

keithwinstein 12y ago
I'm afraid you're mistaken -- frequentist statistics are not a special case of Bayesian statistics, and there is generally no "implicit, built-in prior" that allows you to get frequentist worst-case guarantees from Bayesian techniques. Consider the case of a confidence interval on a binary proportion given a finite number of samples. A frequentist method will produce an interval that includes the true value of the proportion with at least x% probability, even in the worst case, for any proportion between 0 and 1. (E.g. the Blyth-Still-Casella method or the Clopper-Pearson method.) An x% credible interval will include the true value exactly x% of the time, averaged over all values of the proportion weighted according to the prior. This will not provide the same guarantee (much less the same interval extent!), no matter what prior is used. Which is not to say the credible interval is bad or inappropriate. It's just not the same kind of tool. It is optimizing a different penalty function of non-inclusion. (Another example where the "Bayesian" technique is not quite as conservative as necessary to achieve a frequentist-style guarantee, even with a uniform prior: http://www.quora.com/I-have-burned-200-disks-and-I-want-to-make-sure-that-they-are-all-in-perfect-working-order-What-is-the-smallest-size-sample-I-could-test-in-order-to-be-relatively-confident-that-98-of-all-the-disks-are-fine-burned-correctly?share=1 http://www.quora.com/I-have-burned-200-disks-and-I-want-to-m...)

song 12y ago

I always think that having a better understanding of statistics and the mathematics associated with Bayesian calculations would be useful but I don't know where to start. What would be a good book for me to learn?

Matumio 12y ago
Depends on your background. For me Chapter 28 about "Model Comparison and Occam's Razor" in MacKay's book was an eye-opener regarding the usefulness of Bayesian methods. http://www.inference.phy.cam.ac.uk/itprnn/book.html http://www.inference.phy.cam.ac.uk/itprnn/book.html This book is good if you are interested in information theory and machine learning, but it assumes a basic math and probability background.
tmoertel 12y ago
Probability Theory: The Logic of Science, by E.T. Jaynes, is a highly readable definitive reference that builds the theory from the ground up. The first three chapters are all you need for your purposes and are available online for free (link below), but the entire book is wonderful and well worth reading if you are serious about the subject. http://bayes.wustl.edu/etj/prob/book.pdf http://bayes.wustl.edu/etj/prob/book.pdf

bayesianhorse 12y ago

So if two camps of nerds argue, inevitably someone has to submit a post to hackernews "You are both wrong! Let me show you why ..." And maybe this post also rubs me a little wrong because on the one hand I have an interest in bayesian methods and on the other I will never be able to get a thorough nit-picking-able math education...

judk 12y ago

At the end of the day, statistics is about using a rationally consistent process to making decisions under uncertainty, but everyone (except the theoreticians) tries to use it to make correct decisions, conjuring certainty from the ether, and then railing against anyone who disagrees with their choice of articles of faith. In applied statistics we find the bitter religious rivalries of science.

Double_Cast 12y ago

> You prove your understanding by the type of questions you ask. I can't find where I paraphrased this from. Maybe it was Better Explained [0] while explaining the intuition behind complex numbers? As I remember, the context was "Now you understand that complex numbers are rotations through 2 dimensions, I bet a lot of you are asking 'can we extend math to rotations through 3 dimensions?'" (aka quaternions) The vibe I get from this article is "the meaning isn't important, so plug and chug away". No no no no no! Grokking the meaning is important because it allows us to extend our understanding from a strong foundation. E.g. HN a few days ago submitted a paper on "Half Coins"[1] which extends our conventional notion of probability to negative numbers. And after I read the paper, I realized it's not as weird as it sounds. [0] http://betterexplained.com/cheatsheet/ http://betterexplained.com/cheatsheet/ [1] https://news.ycombinator.com/item?id=8187457 https://news.ycombinator.com/item?id=8187457 disclaimer: I don't know anything about statistics.