12 ms·
It is time to stop teaching frequentism to non-statisticians (2012)
- raxxorraxor 4y ago> over-certainty rife among users of statistics. Heh. So people noticed.
- naillo 4y agoThis is an old post but it reminds me of the idea that the blogosphere of the last decade really has moved to arxiv instead. A lot less informal but for technical nuggets of insight that's really the best place with the best culture right now in 2022.
- oldgradstudent 4y ago> The hunt for publishable p-values is nearly always fruitful. If one cannot find a publishable p-value in one’s data—with the freedom to pick and choose models and test statistics, to engage in “sub-group” and sequential analysis, and so on—then one is being lazy. > P-values can and are used to prove anything and everything. The sole limitation is the imagination of the researcher. Fleeting exposure to a 72 × 45 pixel image of the American flag turns one into a Republican; walking through a door (an “event segmentation”) damages your memory; Keynesian theory is wrong; Keynesian theory is right; selenium causes cancer; selenium cures cancer. This list could (and will) go on in perpetuity. Sadly true and well written. However, the author is over certain that using Bayesian statistics will make things better.
- uniqueuid 4y agoRight, people can just go ahead with uniform priors and put significance stars next to credible intervals. But even then, there are a number of benefits, such as even stating CIs in the first place and needing to encounter some literature that's more advanced than clicking "run test" in a readymade GUI.
- RosanaAnaDana 4y agoMaybe not, but the author is pointing at a much broader issue. Reseaerchers are pressured to do as much as possible with as little as possible. So you end up with salad bowl studies that are a mile wide and an inch deep, trying to do everything without even a pilot study to figure out what a minimum sample size might be.
- deleted 4y ago[deleted]
- light_hue_1 4y ago> Civilians just can’t remember that it is forbidden in frequentist theory to talk of the probability of a theory’s or a hypothesis’s truth. Oh. How we all wish the word civilian here din't mean scientists in every field publishing in every journal. And how I wish my reviewers didn't force people to do this :(
- deleted 4y ago[deleted]
- Victerius 4y agoIt's fucking weird. If you're not wearing a military uniform, with rank, you're a civilian. This includes all scientists, except some folks at the NOAA and Public Health Service Commissioned Corps maybe.
- oldgradstudent 4y agoIt's a common metaphor. Civilian is someone who's never been in the armed forced and therefore ignorant of how things really work in the military and in battle. So it's basically a metaphor for a someone who's never studied statistics in depth and therefore ignorant of how statistic really work.
- 1-6 4y agoThis guy is full of himself: “we know par exp ́eriences nombreuses at funestes they are apt to misuse and misinterpret them” It reminds me when I learned about Tyndale translating the Latin Bible into English. There was a pushback in creating a commoner’s version of the Bible where the uneducated would be able to read the Bible for themselves. The Catholic Church was later involved in his death.
- kadoban 4y agoAt least it's Latin you can guess the meaning of from context and such.
- deleted 4y ago[deleted]
- kadoban 4y agoWoops, it's French, should have known.
- lapinot 4y agoYeah, also this is not latin but french, with a typo even (i believe): should be "par expériences nombreuses et funestes" ("et" not "at"). (translation: by numerous and disastrous experiments)
- aeneasmackenzie 4y agoTyndale got in trouble for false translation, not for translation.
- YeGoblynQueenne 4y agoHe is full of himself because he uses a bit of French in an English text? I don't understand how that works. It's a common literary technique to do this and it is not without place in scholarly articles. Just because a text is not pandering to the lowest common denominator with respect to literacy, it's not "full of [itself]".
- 4y ago
- uniqueuid 4y agoI like to use the term "misleading precision" because it's not just frequentism and not just non-statisticians and not just science. That said, every step that forces everybody to acknowledge uncertainty is good. Given our progress e.g. in getting people to recognize how important power is (e.g. since Cohen, 1992 at the latest), I'm very pessimistic. But while we may not be able to get everyone on board the bayes train, we can at least force people to openly show that they don't care about good statistics.
- rwilson4 4y agoBayesian methods have a lot going for them, but you need some way of checking whether the methods are giving sensible answers. Andrew Gelman (well know professor, blogger, and author of Bayesian Data Analysis) relies on simulation, which is a pretty Frequentist thing to do! Bayesian and Frequentist methods are not nearly as at odds as posts like this suggest. Frequentism is mostly about how methods should be evaluated. Bayesianism is mostly about how to incorporate different sources of information. You can assess the Frequentist properties of Bayesian methods! Larry Wasserman wrote a great post about this topic here: https://normaldeviate.wordpress.com/2012/11/17/what-is-bayesianfrequentist-inference/ https://normaldeviate.wordpress.com/2012/11/17/what-is-bayes...
- uniqueuid 4y agoWhile I agree, the most important benefits of bayesian methods - in my opinion - are forcing people to consider (1) uncertainty and (2) effect sizes. These two things alone are a fundamental difference and a major step forward.
- rwilson4 4y agoAt the risk of a no-true-Scotsman argument, I want to point out the difference between Frequentism as practiced by statisticians and non-statisticians. Non-statisticians will simply say that a result was stat sig, without specifying even the corresponding p-value, let alone an effect size point estimate or confidence interval. I always report confidence intervals front and center, and bring in point estimates and p values as supporting characters. And of course discussing to what extent the study design supports causal conclusions.
- lake_vincent 4y agoI also don't like when it's framed as an "either-or" dichotomy. It's like a programming language - there is no "best" one in general, you just have to learn how to choose the best tool for the job at hand, understanding each tool's strengths and weaknesses. It's not that mysterious.
- 4y ago
- civilized 4y agoNonsense. Frequentist statistics is (relatively) easy to understand. If people can't be trusted with it, they certainly can't be trusted with Bayesian statistics. This is more of an argument not to teach anyone statistics at all, other than some unspecified elite that the author feels qualified to learn it. By the way, the author has some interesting obsessions: https://twitter.com/FamedCelebrity/status/1555134947849113600 https://twitter.com/FamedCelebrity/status/155513494784911360... https://twitter.com/FamedCelebrity/status/1555162475863183361 https://twitter.com/FamedCelebrity/status/155516247586318336... https://twitter.com/JohnZmirak/status/1554969385860796420?s=20&t=U_yWO3utLkgsPivYuEHc4Q https://twitter.com/JohnZmirak/status/1554969385860796420?s=... (retweet)
- uniqueuid 4y agoLet me introduce you to Gerd Gigerenzer's paper "The Null Ritual", in which he brilliantly shows that almost nobody understands the rituals of frequentist statistics [1]. "How many students and teachers noticed that all of the statements were wrong? As Figure 1 shows, none of the students did. Every student endorsed one or more of the illusions about the meaning of a p-value. One might think that these students lack the right genes for statistical thinking and are stubbornly resistant to education. A glance at the performance of their teachers, however, indicates that wishful thinking might not be entirely their fault. Ninety percent of the professors and lecturers also had illusions, a proportion almost as high as among their students. Most surprisingly, 80% of the statistics teachers shared illusions with their students." [1] http://library.mpib-berlin.mpg.de/ft/gg/GG_Null_2004.pdf http://library.mpib-berlin.mpg.de/ft/gg/GG_Null_2004.pdf
- civilized 4y agoI don't disagree that many people struggle to properly interpret frequentist statistics. But adopting a Bayesian perspective isn't a magic talisman. The number of people who are competent Bayesians but don't understand frequentist statistics is zero, and no pedagogical changes will improve that.
- uniqueuid 4y ago
- rwilson4 4y agoRegardless of the pros and cons of Bayesian methods, here is what I believe is needed: - Pre-register all studies, declaring sample sizes and power analysis. - Report results regardless of outcome. Eliminate the "we only publish stat sig results" baloney. - Report confidence/credible intervals, adjusting for multiple comparisons as appropriate. Plot the posterior distribution of the effect size if appropriate. - Publish all data and code. - Provide funding for duplicating important studies.
- dalbasal 4y agoAutor doesn't appear to be talking about research/publishing. He just wants frequentism banned for undergrads.
- uniqueuid 4y ago100% this. It's not even hard today, just a cultural shift. I'd add one additional technique: Specification curve analyses. Bonferroni etc. alone won't help against systematic bias in a field and/or misconduct, and specification curves are easy to do, e.g. with specR [1]. [1] https://github.com/masurp/specr https://github.com/masurp/specr
- ngcc_hk 4y agoCan freq. do that and improve as well?
- uniqueuid 4y agoSure. The only thing that's hard is to get a full posterior distribution and to add priors. But the rest can be done in frequentist stats as well. [edit] you can get posteriors with bootstrapping etc. - it's not precisely the same but better than nothing.
- skybrian 4y agoI don't know how often this is practical, but here's an alternative view: preregistration is unimportant when you are looking for enormous effect sizes, and you should do that if you can: https://slimemoldtimemold.com/2022/07/21/on-the-hunt-for-ginormous-effect-sizes/ https://slimemoldtimemold.com/2022/07/21/on-the-hunt-for-gin...
- ogogmad 4y agoWhat about teaching non-parametric statistics, like the use of Concentration Inequalities? See https://scottaaronson.blog/?p=3712 https://scottaaronson.blog/?p=3712 and https://en.wikipedia.org/wiki/Concentration_inequality https://en.wikipedia.org/wiki/Concentration_inequality. Hoeffding's inequality, Chebyshev's inequality, and Chernoff's inequality are broadly applicable, and are therefore less likely to be misused. They also don't require philosophical assumptions about subjective probabilities. In the theory of Multi-Armed Bandit algorithms, compare the Bayesian approach (Thompson sampling) with the Concentration Inequality approach (the UCB algorithm). The former assumes that payoffs are in the 2-element set {0,1}, while the latter allows payoffs to be in the interval [0,1]. I think Bayesian is cool, but it imposes a big burden with choosing sensible priors.
- johnmyleswhite 4y agoThose inequalities often produce confidence intervals that are excessively conservative.
- ogogmad 4y agoProvide details. I don't know what "excessive" means here. They're going to be at least a bit conservative because they apply to more situations, but that's obvious.
- deleted 4y ago[deleted]
- tpoacher 4y agoI feel the opposite. It's time to start teaching it to statisticians again. Properly.
- RosanaAnaDana 4y agoIt should be taught alongside, not in lieu of, other ways of establishing confidence in something. Is a linear model 'better' than a regression developed using a tree method allowed to run over hundreds of parameters? The amount of time I've been forced to take explaining random forest, and why we can trust it's R2 more than we can a linear regression is ridiculous.
- yk 4y agoIf we meet someone identifying as a frequentest then we probably met someone who has thought deeply about philosophy of statistics. By contrast, if we meet someone who identifies as Bayesian, we probably met someone who read a blog post. Bayes theorem tells us, that this doesn't tell us anything about the underlying positions on philosophy of statistics.
- whatever1 4y agoSo far I have observed that statistics as a whole are useless for decision support. They are definitely useful as part of bigger models that we cannot explicitly express them in multiple fields. Physics, chemistry, ML, they all utilize statistics to make useful models and that is great. The issue is that decision making is a singular point, and statistics are very good at not taking any responsibility for their bad predictions. If you make a decision with an expected outcome the statistician will come out swinging. If it was the unexpected outcome then you are faced with “Well you were unlucky, that’s statistics”.
- JoshCole 4y agoThinking inductively is prone to error. This doesn't change if you choose to use a Bayesian approach. If you don't want error, first principles thinking is much more accurate. Some people discover it and think they've discovered the path to thinking without error, but actually first principles don't lead to no error - they lead to a deep and fundamental humility. I find the greatest proponents of Bayesian thinking to be in the rationalist community. So to get across just how wrong everyone is about just about everything pretty much always consider that the pejorative lessons on predictable irrationality and cognitive bias are delusions that come about through the flaws of statistical reasoning. From first principles we know that superhuman chess AI are also predictably irrational and cognitively biased, but that this isn't actually irrational or biased so as to desire us to abandon the choice. Even in chess - much simpler than reality - the correct solution was too large to fit within our universe. It is difficult to articulate the magnitude of this humility. We couldn't fit the chess solution into our universe, but how complex is chess relative to other problems? If the complexity of chess was a grain of sand or even a single atom than it would be an understatement to claim that the more general problems are only as large as our universe.
- e12e 4y agoI came across this post which I found to be a nice recap of the problems with null hypothesis and p values - and overall a more balanced view: 'The "Null Ritual"', Marc Green https://www.visualexpert.com/Resources/nullritual.html https://www.visualexpert.com/Resources/nullritual.html
- kbrkbr 4y agoI tend to think that the problem that underlies all of this is the old question for a sound inductive method, or a deductive method that can replace it. Many Bayesian theoreticists are already named in the comments, for a frequentist view that is well developed and quite strong (i.e. not a strawman) I want to suggest Deborah Mayo’s “Error and the Growth of Empirical Knowledge”. This is related in the following sense to the article: I think the solution for the named ettors may be to stop teaching as if either Bayesianism or Frequentism were set truths and all statistical foundation problems anyway solved, and giving more info on what evidence is and why we think this specific set of tools will help us to prove anything at all. Bayesianism comes with it’s own set of problems, starting with the philosophical underpinnings of it’s flavor of probability, to the often researched theory that humans and scientists are actually not Bayesian thinkers by birth (Giere, Kahneman, Tversky).