6 ms·

Before all that, you need to choose your statistician. My advice is: resuscitate E.T. Jaynes. Failing that, find one of his disciples. Failing that, find a B

Before all that, you need to choose your statistician. My advice is: resuscitate E.T. Jaynes. Failing that, find one of his disciples. Failing that, find a Bayesian. Failing that, read Probability Theory: the Logic of Science, and just do the analysis yourself. Failing that, maybe a Frequentist statistician will do. Maybe.

pja 13y ago
And your Bayesian statistician will acquire their priors how exactly? Bayesian statistics can be very powerful, but it would be a terrible idea to prefer Bayesian approaches to Frequentist ones in all situations.
- loup-vaillant 13y ago
  > And your Bayesian statistician will acquire their priors how exactly? As If Frequentists somehow didn't need priors. Everyone starts with prior knowledge. We might as well use it. Or do you advocate not using every scrap of knowledge available to you? That would be stupid. Sure, prior knowledge can be shaky, or difficult to justify. But at least, a Bayesian will be explicit about it, instead of, like, sweeping normal probability distribution assumptions under the linear regression rug. > it would be a terrible idea to prefer Bayesian approaches to Frequentist ones in all situations. Name three examples that doesn't involve the Frequentist using better prior information than the Bayesian. By the way, Bayesians know that using probability theory correctly is sometimes intractable (combinatorial explosion and all that). In those cases, they will use approximations. But at least, they will know it's an approximation. --- You really should read chapters 1 and 2 of Probability Theory: the Logic of Science. They give a good feel of why Bayesians are correct as a simple matter of fact.
  - rfergie 13y ago
    Those chapters show that any consistent (in some sense) statistical reasoning system must result in the same answers as a Bayesian system. They don't show that the frequentist approach is wrong. If both methods result in the same answer and frequentist methods are easier to use then what is the problem?
    - loup-vaillant 13y ago
      No problem of course. But that's a big if. And how are you going to evaluate whether both methods do in fact deliver the same answer? Also, the Frequentist approach is not wrong. It's inaccurate, to the extent its results differ from the Bayesian ones. This inaccuracy tend to go down as we gather more data. Which is a good thing, or else science itself wouldn't work. One more thing. You said "in some sense". Are you seriously suggesting that the assumptions behind Cox's Theorem can reasonably be challenged?
      - thebooktocome 13y ago
        > Are you seriously suggesting that the assumptions behind Cox's Theorem can reasonably be challenged? Sure: probability is continuous.
        loup-vaillant 13y ago
        You mean, "it is not totally obvious that probability is continuous". Well… To me, it is obvious.
      - rfergie 13y ago
        I'm ashamed to say that I said "in some sense" to be deliberately vague about what I meant by consistency - I didn't want to overstate the strength of Cox's theorem or end up arguing about Godel.
        loup-vaillant 13y ago
        Actually, I was accusing you of understating the strength of Cox's theorem.
  - jtleek 13y ago
    As the author of the guide in question, I feel I should speak up here. Neither frequentist or Bayesian statistics is "just right". It is 100% dependent on the user and whether they use/interpret the quantities correctly. There are both really good and horrible Frequentist and Bayesian statisticians. To imply one or the other is "better" is incorrect and disingenuous. Just my 2 cents.
    - loup-vaillant 13y ago
      Oops. It seems I got carried away. > There are both really good and horrible Frequentist and Bayesian statisticians. Yeah. If I had to choose between Fisher and Anonymous Bayesian, I may chose Fisher. However, unless both kind of statistics yield the same results (I don't think they do), then at least one of them is bogus, by application of the non-contradiction principle. So, while I can imagine there are good Frequentists Statisticians out there, I insist that frequentism itself is bogus.
  - dspeyer 13y ago
    Here's two examples: I'm testing the effectiveness of a drug. Drugs of this class have a certain likelihood of working, the noise in my data is known, the experimental group did this much better than the control... does the drug really work? So far so trivial, in either Bayesianism or Frequentism. Now, I happen to mention that I tested 10000 variants of this drug and only sent data for the one that seemed to work. The rest aren't interesting after all. Under Frequentism, it's easy to take this into account. Under Bayesianism, it requires complex definitions of observations, and is easy to overlook as there's no space for it in the formula. I have a collection of unfair dice. Unfortunately, they all look the same and got dumped on the floor. Now someone grabbed one off the floor at random and wants to make bets with me about it. Even experienced Bayesians are likely to mix up their propositions in a case like this. I say that from having read discussions of similar problems. Yes, if you do it right, it comes out correctly, but Frequentism makes sure you've thought about what you're asking in the same way Bayesianism makes sure you've thought about your priors. Somebody else will have to give a third example. Bayesianism and Frequentism are based on the same math, and math is math. If you use them correctly, they'll get you the same answer every time. The difference is what they make easy, and what mistakes they protect you against.
    - loup-vaillant 13y ago
      First example: of course it's complex (though I have no idea what you mean by "definitions of observations"). The correct answer needs my probability distribution over the algorithm your brain used to select which drug to send to me (my brain already hurts). Also, could you describe the Frequentist method in more detail? I'm sure it must overlook something. Second example: Okay, Bayesian statistics are harder. That's a disadvantage. --- > If you use [Bayesianism or Frequentism] correctly, they'll get you the same answer every time Wat. If they gave invariably the same answer, then, why the endless debates? By the way, here is an apparent factual disagreement bettwen Bayesianism and Frequentism: http://lesswrong.com/lw/ul/my_bayesian_enlightenment/ http://lesswrong.com/lw/ul/my_bayesian_enlightenment/
    - landismj 13y ago
      Third example: A researcher has millions of datasets to analyze, with each dataset containing enough data points such that frequentist asymptotics are satisfied. You are tasked with finding summary statistics for all datasets. The maximum likelihood and maximum a posterior (MAP) estimators are equal within some tolerance for a subset of these datapoints. However, the marginal likelihood function is computationally intractable, so the Bayesian must use expensive methods to produce MAP estimates, e.g. using Markov chain Monte Carlo (MCMC). For complex posterior distributions, MCMC requires careful programming and verification procedures, which can be prohibitive in practice. There are very many real-world problems that have fast and accurate frequentist solutions, but slow and difficult Bayesian solutions. Despite my personal bias -- my research primarily relies on Bayesian inference -- I can't fathom how one can reasonably argue that frequentist approaches are always inferior, even in applied statistics.
- capnrefsmmat 13y ago
  You may be very interested to read about Bayes factors, which allow interpretable hypothesis tests without prespecified priors: https://www.andrew.cmu.edu/user/kk3n/simplicity/KassRaftery1995.pdf https://www.andrew.cmu.edu/user/kk3n/simplicity/KassRaftery1... (though one does have to specify the hypotheses to be tested more exactly than, say, "the effect is not zero.")
sanskritabelt 13y ago
On the other hand, if you let yourself update your prior with the actual data, you'll see that frequentist statistics has had the better part of a century worth of successes.
- loup-vaillant 13y ago
  The same could be said of Newtonian mechanics. It did not stop it from being completely dead by the 1930's. Both Newtonian mechanics and Frequentists statistics are approximations. Which makes them both useful, and inaccurate (how inaccurate depends on the situation at hand).
  - sanskritabelt 13y ago
    Just out of curiosity, what kinds of statistical problems do you tend to solve, or are you just a fanboy? I saw you posted a link to something on lesswrong, which is confusing an application of Bayes Rule with Bayesian statistics. Maybe you should read what an actual statistician has to say on the matter: http://normaldeviate.wordpress.com/2012/11/17/what-is-bayesianfrequentist-inference/ http://normaldeviate.wordpress.com/2012/11/17/what-is-bayesi... instead of a Harry Potter fanfic author.
    - loup-vaillant 13y ago
      Fanboy. I have read half of E.T. Jaynes' Probability Theory: the Logic of Science so far. I have read the link you speak of. I have already responded here: http://normaldeviate.wordpress.com/2012/11/17/what-is-bayesianfrequentist-inference/#comment-2793 http://normaldeviate.wordpress.com/2012/11/17/what-is-bayesi... I don't think my LessWrong link confuses Bayes' Rule with Bayesian statistics. Why do you think Eliezer responded 1/2 to the brain teaser? He does not say it, but I'm pretty sure he just assumed that a mathematician who has 2 boys is twice as likely to spontaneously say "I have at least one boy", compared to a mathematician who has only one boy. The disagreement between his inference (which he did not know was "Bayesian" at the time) and "Orthodox" statistics didn't come from Bayes' Rule. It came from the use of a non-uniform prior to begin with. Which Frequentism rejects, because it's "subjective". So, instead of using this highly relevant prior information, it just uses a uniform prior. (By the way, this is nuts. A scientist should never throw away relevant information.) Both methods then use Bayes' Rule. They just start with different priors: best guess and objective-looking, respectively. (I'm still wondering why anyone would use antything but one's best guess.)

pja 13y ago

And your Bayesian statistician will acquire their priors how exactly? Bayesian statistics can be very powerful, but it would be a terrible idea to prefer Bayesian approaches to Frequentist ones in all situations.

loup-vaillant 13y ago
> And your Bayesian statistician will acquire their priors how exactly? As If Frequentists somehow didn't need priors. Everyone starts with prior knowledge. We might as well use it. Or do you advocate not using every scrap of knowledge available to you? That would be stupid. Sure, prior knowledge can be shaky, or difficult to justify. But at least, a Bayesian will be explicit about it, instead of, like, sweeping normal probability distribution assumptions under the linear regression rug. > it would be a terrible idea to prefer Bayesian approaches to Frequentist ones in all situations. Name three examples that doesn't involve the Frequentist using better prior information than the Bayesian. By the way, Bayesians know that using probability theory correctly is sometimes intractable (combinatorial explosion and all that). In those cases, they will use approximations. But at least, they will know it's an approximation. --- You really should read chapters 1 and 2 of Probability Theory: the Logic of Science. They give a good feel of why Bayesians are correct as a simple matter of fact.
- rfergie 13y ago
  Those chapters show that any consistent (in some sense) statistical reasoning system must result in the same answers as a Bayesian system. They don't show that the frequentist approach is wrong. If both methods result in the same answer and frequentist methods are easier to use then what is the problem?
  - loup-vaillant 13y ago
    No problem of course. But that's a big if. And how are you going to evaluate whether both methods do in fact deliver the same answer? Also, the Frequentist approach is not wrong. It's inaccurate, to the extent its results differ from the Bayesian ones. This inaccuracy tend to go down as we gather more data. Which is a good thing, or else science itself wouldn't work. One more thing. You said "in some sense". Are you seriously suggesting that the assumptions behind Cox's Theorem can reasonably be challenged?
    - thebooktocome 13y ago
      > Are you seriously suggesting that the assumptions behind Cox's Theorem can reasonably be challenged? Sure: probability is continuous.
      - loup-vaillant 13y ago
        You mean, "it is not totally obvious that probability is continuous". Well… To me, it is obvious.
    - rfergie 13y ago
      I'm ashamed to say that I said "in some sense" to be deliberately vague about what I meant by consistency - I didn't want to overstate the strength of Cox's theorem or end up arguing about Godel.
      - loup-vaillant 13y ago
        Actually, I was accusing you of understating the strength of Cox's theorem.
- jtleek 13y ago
  As the author of the guide in question, I feel I should speak up here. Neither frequentist or Bayesian statistics is "just right". It is 100% dependent on the user and whether they use/interpret the quantities correctly. There are both really good and horrible Frequentist and Bayesian statisticians. To imply one or the other is "better" is incorrect and disingenuous. Just my 2 cents.
  - loup-vaillant 13y ago
    Oops. It seems I got carried away. > There are both really good and horrible Frequentist and Bayesian statisticians. Yeah. If I had to choose between Fisher and Anonymous Bayesian, I may chose Fisher. However, unless both kind of statistics yield the same results (I don't think they do), then at least one of them is bogus, by application of the non-contradiction principle. So, while I can imagine there are good Frequentists Statisticians out there, I insist that frequentism itself is bogus.
- dspeyer 13y ago
  Here's two examples: I'm testing the effectiveness of a drug. Drugs of this class have a certain likelihood of working, the noise in my data is known, the experimental group did this much better than the control... does the drug really work? So far so trivial, in either Bayesianism or Frequentism. Now, I happen to mention that I tested 10000 variants of this drug and only sent data for the one that seemed to work. The rest aren't interesting after all. Under Frequentism, it's easy to take this into account. Under Bayesianism, it requires complex definitions of observations, and is easy to overlook as there's no space for it in the formula. I have a collection of unfair dice. Unfortunately, they all look the same and got dumped on the floor. Now someone grabbed one off the floor at random and wants to make bets with me about it. Even experienced Bayesians are likely to mix up their propositions in a case like this. I say that from having read discussions of similar problems. Yes, if you do it right, it comes out correctly, but Frequentism makes sure you've thought about what you're asking in the same way Bayesianism makes sure you've thought about your priors. Somebody else will have to give a third example. Bayesianism and Frequentism are based on the same math, and math is math. If you use them correctly, they'll get you the same answer every time. The difference is what they make easy, and what mistakes they protect you against.
  - loup-vaillant 13y ago
    First example: of course it's complex (though I have no idea what you mean by "definitions of observations"). The correct answer needs my probability distribution over the algorithm your brain used to select which drug to send to me (my brain already hurts). Also, could you describe the Frequentist method in more detail? I'm sure it must overlook something. Second example: Okay, Bayesian statistics are harder. That's a disadvantage. --- > If you use [Bayesianism or Frequentism] correctly, they'll get you the same answer every time Wat. If they gave invariably the same answer, then, why the endless debates? By the way, here is an apparent factual disagreement bettwen Bayesianism and Frequentism: http://lesswrong.com/lw/ul/my_bayesian_enlightenment/ http://lesswrong.com/lw/ul/my_bayesian_enlightenment/
  - landismj 13y ago
    Third example: A researcher has millions of datasets to analyze, with each dataset containing enough data points such that frequentist asymptotics are satisfied. You are tasked with finding summary statistics for all datasets. The maximum likelihood and maximum a posterior (MAP) estimators are equal within some tolerance for a subset of these datapoints. However, the marginal likelihood function is computationally intractable, so the Bayesian must use expensive methods to produce MAP estimates, e.g. using Markov chain Monte Carlo (MCMC). For complex posterior distributions, MCMC requires careful programming and verification procedures, which can be prohibitive in practice. There are very many real-world problems that have fast and accurate frequentist solutions, but slow and difficult Bayesian solutions. Despite my personal bias -- my research primarily relies on Bayesian inference -- I can't fathom how one can reasonably argue that frequentist approaches are always inferior, even in applied statistics.
capnrefsmmat 13y ago
You may be very interested to read about Bayes factors, which allow interpretable hypothesis tests without prespecified priors: https://www.andrew.cmu.edu/user/kk3n/simplicity/KassRaftery1995.pdf https://www.andrew.cmu.edu/user/kk3n/simplicity/KassRaftery1... (though one does have to specify the hypotheses to be tested more exactly than, say, "the effect is not zero.")

sanskritabelt 13y ago

On the other hand, if you let yourself update your prior with the actual data, you'll see that frequentist statistics has had the better part of a century worth of successes.

loup-vaillant 13y ago
The same could be said of Newtonian mechanics. It did not stop it from being completely dead by the 1930's. Both Newtonian mechanics and Frequentists statistics are approximations. Which makes them both useful, and inaccurate (how inaccurate depends on the situation at hand).
- sanskritabelt 13y ago
  Just out of curiosity, what kinds of statistical problems do you tend to solve, or are you just a fanboy? I saw you posted a link to something on lesswrong, which is confusing an application of Bayes Rule with Bayesian statistics. Maybe you should read what an actual statistician has to say on the matter: http://normaldeviate.wordpress.com/2012/11/17/what-is-bayesianfrequentist-inference/ http://normaldeviate.wordpress.com/2012/11/17/what-is-bayesi... instead of a Harry Potter fanfic author.
  - loup-vaillant 13y ago
    Fanboy. I have read half of E.T. Jaynes' Probability Theory: the Logic of Science so far. I have read the link you speak of. I have already responded here: http://normaldeviate.wordpress.com/2012/11/17/what-is-bayesianfrequentist-inference/#comment-2793 http://normaldeviate.wordpress.com/2012/11/17/what-is-bayesi... I don't think my LessWrong link confuses Bayes' Rule with Bayesian statistics. Why do you think Eliezer responded 1/2 to the brain teaser? He does not say it, but I'm pretty sure he just assumed that a mathematician who has 2 boys is twice as likely to spontaneously say "I have at least one boy", compared to a mathematician who has only one boy. The disagreement between his inference (which he did not know was "Bayesian" at the time) and "Orthodox" statistics didn't come from Bayes' Rule. It came from the use of a non-uniform prior to begin with. Which Frequentism rejects, because it's "subjective". So, instead of using this highly relevant prior information, it just uses a uniform prior. (By the way, this is nuts. A scientist should never throw away relevant information.) Both methods then use Bayes' Rule. They just start with different priors: best guess and objective-looking, respectively. (I'm still wondering why anyone would use antything but one's best guess.)