5 ms·
I don't like this cartoon
- colechristensen 14y agoWhile it's true (and should be obvious) that a competent statistician wouldn't make such silly mistakes, there are quite a few scientists who are not-so-competent statisticians who publish results with those exact errors.
- Evbn 14y agoIncluding the entire A/B testing world, from startup to Amazon.com and big banks. (Source: working with all three.)
- InclinedPlane 14y agoInteresting, but it IS a joke, which usually requires taking some liberties with the truth. For example, an issue I have with it is that it's not actually possible for the Sun to explode. It is incapable of going nova. This is an old sci-fi trope, of a civilization that dies because its Sun goes nova, but this isn't a thing that actually happens. There are very specific circumstances where a nova can occur, and it is typically a recurring thing. More so, it requires the presence of a white dwarf star, which must come into existence only after a star has gone into a red giant phase. None of these things are conducive to life developing on nearby planets, they aren't even conducive to the continued existence of nearby planets. Regardless, our star is not a white dwarf, nor does it have a companion and therefore will never produce a nova. More so, it is about 1/8th too small to go supernova.
- timbre 14y agoThat's the point. The Bayesian incorporates this information into the prior. ;)
- btilly 14y agoI learned something today. http://en.wikipedia.org/wiki/Helium_flash http://en.wikipedia.org/wiki/Helium_flash is what I thought was the process that leads to novas, it is not. That said, from our point of view a helium flash is pretty scary. Of course the Sun is not expected to do this for another 5 billion years. But according to http://www.whillyard.com/science-pages/our-solar-system/sun-evolution.html http://www.whillyard.com/science-pages/our-solar-system/sun-... the Earth is likely to be uninhabitable in about a billion years anyways.
- SparrowOS 14y agoGod says... and in the night is as a thief. 24:15 The eye also of the adulterer waiteth for the twilight, saying, No eye shall see me: and disguiseth his face. 24:16 In the dark they dig through houses, which they had marked for themselves in the daytime: they know not the light. 24:17 For the morning is to them even as the shadow of death: if one know them, they are in the terrors of the shadow of death. 24:18 He is swift as the waters; their portion is cursed in the earth: he beholdeth not the way of the vineyards. 24:19 Drought and heat consume the snow waters: so doth the grave those which have sinned.
- pav3l 14y agoI really like this sentence: > All statisticians use prior information in their statistical analysis. Non-Bayesians express their prior information not through a probability distribution on parameters but rather through their choice of methods. Also, from the comments: >My problem is ultimately not with the cartoon. My problem is that there are practitioners and teachers of statistics who spread cartoonish ideas about statistical methods without recognizing these ideas are inaccurate.
- Symmetry 14y agoI'm not sure how it can be a good idea to be choosing from various statistical methods based on which one will give you the sort of answer you intuitively think is correct. I mean, if you're going to bite the bullet and bring priors into your analysis in an ad-hoc way like that, why not just acknowledge their existence mathematically?
- pav3l 14y agoWhen you are reporting, say, a p-value from your ANOVA F-test, you are making formal mathematical assumptions, such as normal marginal distribution of your dependent variable. A lot of Frequentist methods (hypothesis testing) are really just mathematical shortcuts from times when computation was more expensive. The problem is, many people tend to misuse the tests where they are not appropriate either because those give "better" answers, or simply out of ignorance.
- Symmetry 14y agoRight, but the way you deal with the situation in the comic is going to end up being much fuzzier and more subjective issue of picking you reference classes.
- zzzeek 14y agoJust that response seems to prove the cartoon's point.
- jme3 14y agoI don't see how, really. Gelman himself is a (rather famous) Bayesian, and if you read the comments, you'll see that Randall himself pops up and basically cedes Gelman's point.
- lisper 14y agoSince it's Sunday and I'm in procrastination mode I took a little excursion into the rabbit hole and found this: http://www.stat.columbia.edu/~gelman/research/published/feller8.pdf http://www.stat.columbia.edu/~gelman/research/published/fell... The money quote: "More than that, though, the big, big problem with the Pr(sunrise tomorrow | sunrise in the past) argument is not in the prior but in the likelihood, which assumes a constant probability and independent events. Why should anyone believe that? Why does it make sense to model a series of astronomical events as though they were spins of a roulette wheel in Vegas? Why does stationarity apply to this series? That’s not frequentist, it isn’t Bayesian, it’s just dumb."
- zanny 14y agoIf the default state of procrastination was researching higher order mathematical theory the world would be.... probably more predictable. I'll let myself out.
- jere 14y agoWow, this is embarrassing. I took statistics, but it's been almost a decade. I got a pretty nice review on Bayes' theorem in an AI class I took last year though. I thought I understood the xkcd comic quite clearly, but I'm completely lost on the posted article. Apparently, Munroe was a bit confused also. >The truth is, I genuinely didn’t realize Frequentists and Bayesians were actual camps of people—all of whom are now emailing me. http://andrewgelman.com/2012/11/16808/#comment-109366 http://andrewgelman.com/2012/11/16808/#comment-109366
- acomar 14y agoSome how this only makes the comic and reaction way more funny.
- StavrosK 14y agoI don't understand how the two camps can exist. Don't the two methods produce different results? Surely, only one is true. Which is it?
- jme3 14y agoThe point of Gelman's reply is that the comic is actually comparing a Bayesian to an absurdly incompetent Frequentist, so there' really no conflict. No (modestly intelligent) Frequentist would mis-apply this methodology in this circumstance.
- StavrosK 14y agoSorry, I wasn't talking about the comic. In general, don't the two approaches give different results? Surely, only one is "correct".
- michaelhoffman 14y agoThey are both models and as such, you might consider that neither of them are "correct." But they are both useful, sometimes in different circumstances. "Essentially, all models are wrong, but some are useful." — George Box
- StavrosK 14y agoI see, thank you all very much for the clarifications.
- equark 14y agoBoth are correct but they target different things. The disagreement is around what is the target should be and the advantages and disadvantages of choosing these targets. Bayesians are interested in p(unknown|data) and frequentists are interested in p(data|unknown = H0). Inference can be framed either way but means different things.
- 14y ago
- deleted 14y ago[deleted]
- amouat 14y agoReminds me of the letter Babbage wrote to Tennyson, correcting his poem: "In your otherwise beautiful poem one verse reads, Every moment dies a man, Every moment one is born If this were true the population of the world would be at a standstill. In truth, the rate of birth is slightly in excess of that of death. I would suggest that the next edition of your poem should read: Every moment dies a man Every moment 1 1/16 is born Strictly speaking the actual figure is so long I cannot get it into a line, but I believe the figure 1 1/16 will be sufficiently accurate for poetry." Seems jgc has written a blog about it: http://blog.jgc.org/2010/09/on-being-nerd.html http://blog.jgc.org/2010/09/on-being-nerd.html
- praptak 14y agoActually a moment is an infinitesimally small amount of time, therefore in order to model the birth/death distribution across moments we need to rigorously define... wait, what am I doing with my life?!
- mannjani 14y agoCan't we all just take a cartoon AS a cartoon? I don't think Munroe is trying to'smack down' frequentists here, it's all just for a little fun. Look at his own comment on the post: http://andrewgelman.com/2012/11/16808/#comment-109366 http://andrewgelman.com/2012/11/16808/#comment-109366
- lcargill99 14y agoIsn't the joke that if the sun did just go nova, then a $50 bet is meaningless?
- DaniFong 14y agoI do like this cartoon; it's a very nice representation of the problem with the viewpoint. The main issue is that it make frequentist humans look like idiots. They're not. Simply oftentimes lead astray -- as this example illustrates. In particular, the epistemological fact that people misuse p-values must be exposed. Just because it is unlikely something happened by chance, in one way, in your model, does not prove your hypothesis correct.
- keithwinstein 14y agoHere is my writeup on the "real" difference between frequentist and Bayesian methods: http://stats.stackexchange.com/a/2287/1122 http://stats.stackexchange.com/a/2287/1122 Even more here: http://qr.ae/17BEW http://qr.ae/17BEW The truth is they both make tradeoffs that can appear ridiculous. In fact, the criticisms of confidence intervals and p-values apply almost exactly, in transpose, to credibility intervals and posterior probabilities. Confidence intervals and p-values are a worst-case technique. The p-value will always control the false positive rate below alpha, even in the worst input. Sometimes you do want this -- e.g. when we say that the worst-case runtime of QuickSort is O(n^2), that's useful, even if we do have a prior distribution over the inputs and could also say that the expected runtime is O(n log n). But the errors are correlated across observations. You can have a valid "95%" confidence interval that always produces total nonsense when the experiment ends up with output x, as long as x happens <5% of the time for all possible inputs. Credibility intervals and posterior probabilities are an average-case technique, where we integrate over the prior. Even if the prior is correct, the errors are correlated across inputs, which can be a problem. In the cookie-jar example at stackexchange, the 70% credibility interval is "wrong" 80% of the time when the jar is type B. That means if you send out 100 "Bayesian" robots to assess what type of jar you have, each robot sampling one cookie, you will expect 80 of the robots to get the wrong answer, each having >73% posterior probability in that wrong conclusion! That's a problem, especially if you want most of the robots to agree on the right answer. The two methods just make different tradeoffs in the way they quantify uncertainty. My quibble with the cartoon, though, was that it's not really about the frequentist vs. Bayesian debate. If you want to decide whether to take action (like shuttering a satellite) in response to a "YES" output from the instrument, everybody will agree that you need to calculate {rate of events} * {false negative rate} * {cost of false negative} and compare it with {1 - rate of events} * {false positive rate} * {cost of false positive}. The frequentist agrees with this math, the Bayesian agrees with this math, and the math doesn't even use Bayes' rule. This is basic actuarial science or decision theory. The frequentist might do the mechanics in a certain way. They may say they are first going to calculate a p-value, and then ask whether the p-value is less than a threshold alpha, where alpha was set based on the costs and rate of events in order to control the "false discovery rate." And then take action only on a "significant" result. The Bayesian might do the calculation a little differently too; they could say they are first going to use the expected rate of events as a prior, then calculate the conditional probability that there has been an event (given the instrument's reading), and then multiply this posterior probability by the cost of false negative, and its complement by the cost of false positive, to decide which action has lower expected cost. But both the frequentist and Bayesian will get the same answer and end up with the same result as somebody who evaluates the inequality above directly. I don't think any technique has a monopoly on the correct answer here.
- mistercow 14y agoHere's the thing: the frequentist in the comic has made an error even by frequentist standards, and that error is equivalent, in Bayesian thinking, to choosing an inappropriate prior. The problem is that many frequentist techniques implicitly choose a prior for you. That's handy since choosing an appropriate prior is hard. But it also abstracts away the choice of prior. If a Bayesian makes this mistake, anyone can look at their math and say "There. That's where you chose a bad prior." If a frequentist makes this mistake, you have to have a complicated analysis to explain why the method used is inappropriate.
- gajomi 14y agoI agree in with this analysis. The cartoon is, of course, a caricature but highlights the nature of the essential problem you point out, which is that unraveling the line of reasoning carved out by a frequentist is not a straightforward matter, even if the argument is correct. From the Bayesian perspective all arguments are deductive once you have all the information. A typical reply to this observation is that in most cases it will be clear how the steps of a calculation translate into the various assumptions needed to clarify the argument. Of course, what counts as "typical" depends on the kinds of questions one asks. If there are only a few kinds of problems in your field, then maybe you can get away with heuristic lines of reasoning, patching up problem in special cases as needed. But if you are, say, trying to write a general purpose software for statistical analysis it will not be possible to rely on inductive lines of reasoning particular to a certain field. This might partially explain the popularity of the Bayesian approach in machine learning circles.
- rcthompson 14y agoGiven that Randall Munroe has already expressed that the Frequentist vs. Bayesian angle of the comic was an afterthought, I think the real point (and the point that I actually took away from the comic) is about the blind application by scientists of the 0.05 P-value threshold for significance without regard to the specific circumstances of the experiment, which I can attest is a huge issue in the scientific literature. Another huge issue is using a statistical test on data that is known not to satisfy the assumptions of the test, either out of ignorance or because the test gives a good (i.e. P<0.05) result where the correct test doesn't.
- Evbn 14y agoCf Randall's own classic comic about jellybeans causing cancer, of maybe one color of jellybeans causing cancer.
- Evbn 14y agoThis cartoon actually explains why the Higgs Boson and tachyon hunting folks use p=0.000001 or so, not p=0.05. Extraordinary claims require extraordinary evidence. Not a problem with frequentism at all.