5 ms·
Frequentist Statistics are Frequently Subjective
- smanek 17y agoTo my (limited) understanding, Bayesian statistics are just as subjective. If you and I start with different priors, and are then are fed the same evidence, we can end up with vastly different conclusions. Since priors are defined as a function of my state of mind (or state of ignorance, as it were), it seems pretty subjective that the 'result' should depend so heavily on my initial state of mind. E.T. Jaynes' book has a example of this with regard to E.S.P. research (I believe http://omega.albany.edu:8008/ETJ-PS/cc5d.ps http://omega.albany.edu:8008/ETJ-PS/cc5d.ps is the relevant chapter - but it's been a year or two since I've read the book). A real statistician should feel free to correct me though - I'm more of an algebra guy ...
- mnemonicsloth 17y agoTo my (limited) understanding, Bayesian statistics are just as subjective. I'm still reading, but that seems to be the point. As Yudkowsky puts it, "probability is ignorance and ignorance is a state of mind"
- yummyfajitas 17y agoThe argument is that Bayesian statistics are less subjective because while we won't come to the same conclusion, we will perform the same update to our prior probabilities. An example: consider a medical diagnostic test with a 1% false positive rate and no false negatives. If I understand him correctly, Eliezer is arguing that the scientist (i.e. the lab running the test) should present an update rule, not the posterior probability. I.e., the lab report should say: "Whatever your prior estimate of P(sick) is, your posterior should now be P(sick)/(0.01 P(not sick) + P(sick))" instead of saying "the patient has an x% chance of being sick". Suppose before the test I believe the odds the patient is sick are 1% and you think they are 10%. Once the test comes out positive, I've updated my probability estimate to 50%, you to 92%. The update rule provided by Bayesian statistics is objective, even if the posterior is subjective. The frequentists have nothing comparable which is objective. See also this link: http://www.overcomingbias.com/2009/02/share-likelihood-ratios-not-posterior-beliefs.html http://www.overcomingbias.com/2009/02/share-likelihood-ratio...
- smanek 17y agoAh, that makes sense - thanks.
- jules 17y agoThe difference is that bayesians acknowledge this explicitly. You publish the way to calculate the posterior given someones prior so that everyone can fill in their own priors to determine their posteriors. This calculation is objective. With frequentist statistics everything is subjective, including the results you publish.
- btilly 17y agoYour understanding is incomplete. Bayesian statistics are not subjective at all. Instead they objectively describe the correlation between prior belief and conclusion. People may draw different conclusions for subjective reasons, but that subjectivity is in the people, not the statistics. Now it seems you are objecting to the fact that your initial state of mind affects your conclusion. But it is unavoidable that there is no way to make sense of observation except through the lens of prior beliefs. You can be explicit about it and draw conclusions with a correct methodology about it as in Bayesian statistics. You can be implicit about it and draw conclusions with an incorrect methodology as in frequentist statistics. (See my other post in this thread for an explanation of why frequentist methodology is incorrect.) Let me offer a simple example. Suppose a pregnant woman you know gives birth to a boy. Is that evidence that babies are more likely to be boys than girls? Obviously it is. Is it strong evidence? Obviously not. Should we upon observing that conclude that boys are more likely than girls? Obviously not. Now suppose that you have no prior knowledge other than we see about similar numbers of boys and girls. I know that of the last 100,000 babies born in the USA, 51,157 are boys. Suppose we are both told that at the local hospital, 92 of the last 200 babies born were boys. I submit that we both will and should wind up with different conclusions about the relative likelyhood of boys and girls. Why? Because different prior knowledge lead to different prior beliefs, and those prior beliefs when modified by identical evidence lead to different conclusions.
- smanek 17y agoYou have to admit that your example is pretty contrived though - usually setting priors isn't so clean. How do you 'properly' set the prior probability that someone really has E.S.P.? That a researcher is secretly colluding with a test subject? That a researcher is flat out fabricating their results? That this is just an instance of the Hans effect? Or a thousand other possible hypotheses ...
- btilly 17y agoI picked my example so that the proper influence of the prior would be clear. You are absolutely right that there are many cases where different people with equivalent information have different beliefs. But being faced with complications like this makes correct reasoning more important, and not less. When you try to sweep the proper influence of the prior under the carpet, people will manipulate their statistics to draw the conclusions that they want. Worse still, if each is using subjective techniques that they believe are objective, the unexamined assumption will lead to them talking past each other. By contrast with correct reasoning you can show each why they continue to believe what they believe while showing them how the other side is not going to have their opinions changed. In my experience this opens up a bigger possibility of useful dialog and changed opinions.
- btilly 17y agoThe real problem with statistics is that people want an answer to an impossible problem. Namely they want to be told the probability of the world being a particular way. But anyone familiar with conditional probability can easily see that there is no way to come up with that answer, because the conditional probability of something after an observation depends on the assumed probability before the observation. There are multiple approaches to this problem. What frequentist statistics does is answers a different question. Namely, "What is the probability of getting a result this strongly against the null hypothesis if the null hypothesis is true." This is appealing in that it is an objective probability that seems to say something about the problem under discussion. However people consistently read it as, "What is the probability that the null hypothesis is true?" Which is simply wrong. There is a second problem with frequentist statistics, which is what this article is about. Which is that the objective-looking probability you get depends on the null hypothesis chosen in ways it shouldn't. Basic familiarity with Bayes' Theorem and conditional probability demonstrates that it is impossible for it to matter whether your intention was to flip a coin 6 times, flip it until you get both heads and tails, or flip it until you get heads. This factor cannot affect the conditional probability of the coin being biased. But those three different intentions translate into 3 different calculations, and 3 different answers in frequentist statistics. So that's the frequentist approach. What is the Bayesian approach? It is to come up with statistics that inform us on how prior beliefs before observation turn into posterior beliefs after observation. The advantage of this method is that it is intellectually honest. The disadvantages are that it is complicated and people notice that it is confusing. (The frequentist approach is confusing as well, but people don't notice their confusion. Instead the confidently draw the wrong conclusion that the null hypothesis has been proven.) There are other approaches. The article touched on my favorite when it pointed out that we should report likelyhood ratios rather than probabilities. This is absolutely right. The effect of an observation is to modify our prior beliefs, and likelyhood ratios concisely describe how we should modify them. Plus they have the great ability to stack - you can take 3 experiments and combine their likelyhood ratios to come up with the likelyhood ratio for having seen all three experiments. Unfortunately, though, everyone knows frequentist methods, people accept them, and it is very hard to get people to see what is wrong with them. So alternate approaches, though theoretically superior, face an uphill battle towards acceptance.
- gort 17y agoTo me, more interesting than Bayesianism versus Frequentism was the observation that scientific procedure as it currently stands allows a field like Parapsychology to survive.
- mattmcknight 17y agoI get lost in the argument with these simple coin flipping examples. The way it's presented is so confusing for the weird second case of how many trials it takes to get a tail. In the first case, the probability of getting "five tails or more" from a fair coin is 11%, while in the second case, the probability of a fair coin requiring "at least five tails before seeing one heads" is 3%. I didn't check the numbers here, but this seems perfectly reasonable to me in terms of hypothesis testing. He's playing on the fact that for cases like THXXXX you would get a result of 2 in the second experiment. You are taking the same results, but applying a different metric to them (position of first head versus count of tails). Of course p will be different. I understand that the value chosen for significance of p is subjective, but the experiment itself makes perfect sense to me.
- btilly 17y agoThe problem is that the conditional probability of the coin being biased given the observation of TTTTTH absolutely does not and cannot depend one whether you intended to flip until you saw a heads, or were going to flip 6 times. Therefore you've got a subjective factor influencing your conclusion that probability theory says should not be a factor if you are reasoning consistently.
- hamilton 17y agoThis debate has been raging for 250 years. Brad Efron, one of the greatest living statisticians today and one of the few who can transcend the debate, has VERY interesting things to say about it. He believes that Empirical Bayes, or using some of your data to model your prior, will win the day. http://www-stat.stanford.edu/~ckirby/brad/papers/2005NEWModernScience.pdf http://www-stat.stanford.edu/~ckirby/brad/papers/2005NEWMode...
- nova 17y ago> or using some of your data to model your prior I think that kinda missing the point.
- hamilton 17y agoI beg to differ (not sure if you actually read the Efron talk I just posted). I get the thrust of the Less Wrong article, though I think that frankly his language and vitriol is direly misplaced. He's railing against some class of Frequentists who attack Bayesians. This is sort of a Fox News tactic; pick someone, in this case, a commenter on a blog, and generalize their comment to the entire population. It's fairly unstatistical, if you ask me. I don't get why anyone would do this. A good Frequentist is not a lost idiot, nor does she rail against Bayesians. In fact, a good Frequentist understands the beauty of Bayesian methods, too. I'd even wager that a good Bayesian understands why Frequentist methods have dominated the 20th century. I'm attempting to realize that rare event in these vitriolic philosophical academic debates - add a bit of middle-groundness. In this case, the middle ground of Empirical Bayes has proven to be very, very useful in large-scale simultaneous inference problems. It's proof that there IS such thing as convergence in this debate. Taking the shrill tone the author takes as a signal of how little regard the two sides have for each other, it's obvious that we need some convergence.
- tel 17y agoI don't think an upvote does this enough justice. At the end of the day, we're all just trying to be better. Shaking strawmen at progress just because it's not perfect (you'll note that some of the Less Wrong/Overcoming Bias arguments leave small disclaimers of assumptions of infinite computing power; not all, but some) is just as foolish as refusing to see when the ground moves beneath your feet.
- deleted 17y ago[deleted]