7 ms·

It is time to stop teaching frequentism to non-statisticians (2012)

• NewsaHackO 1y ago
  It’s weird how random people can submit non peer reviewed articles to preprint repos. Why not just use a blog site, medium or substack?
  • jxjnskkzxxhx 1y ago
    > Why not just use a blog site, medium or substack? Because it looks more credible, obviously. In a sense it's cargo cult science: people observe this is the style of science, and so copy just the style; to a casual observer it appears to be science.
    • nickpsecurity 1y ago
      Professional science has been doing that a long time if one considers that many published works were never independently tested and replicated. If it's a scientist, and uses scientific descriptions, many just repeat it from there.
      • jxjnskkzxxhx 1y ago
        Overly reductionistic. At the same time a proper rebuttal isn't worth the time for someone who's clearly not looking to understand.
    • NeutralCrane 1y ago
      Publication in a journal is not a requirement for the scientific method. If anything, the insistence that something not published in a scientific journal is not science is, itself, cargo cult scientism.
  • billfruit 1y ago
    Why the gatekeeping. Only what is said matters, not who says it.
    • BlarfMcFlarf 1y ago
      Peer review specifically checks that what is being said passes scrutiny by experts in the field, so it is very much about what is being said.
      • SJC_Hacker 1y ago
        They why isn't it double blind ?
        • BDPW 1y ago
          Often reviewing is executed double blind for exactly this reason. This can be difficult in small fields where you can more-or-less guess who's working on what, but the intent is definitely there.
        • mcswell 1y ago
          I've reviewed computational linguistics papers in the past (I'm retired now, and the field is changing out from under me, so I don't do it any more). But all the reviews I did were double blind.
    • tsimionescu 1y ago
      That's a cute fantasy, but it doesn't work beyond a tiny scale. Credentials are critical to help filter data - 8 billion people all publishing random info can't be listened to.
      • SoftTalker 1y ago
        > 8 billion people all publishing random info can't be listened to. Yet it's what we train LLMs on.
  • groceryheist 1y ago
    Two reasons: 1. Preprint servers create DOIs, making works better citable. 2. Preprint servers are archives, ensuring works remain accessible. My blog website won't outlive me for long. What happened to geocities could also happen to medium.
    • SoftTalker 1y ago
      Who would want to cite a random unreviewed preprint?
      • mitthrowaway2 1y ago
        You don't get a free pass to not cite relevant prior literature just because it's in the form of an unreviewed preprint. If you're writing a paper about a longstanding math problem and the solution gets published on 4chan, you still need to cite it.
        • NooneAtAll3 1y ago
          tbf, you cite the paper that described and discussed said solution in the more appropriate form
          • mousethatroared 1y ago
            You cite the form you encountered and if you're any good of a researcher you will have encountered the original 4chan anon post, Borges' short story, or Chomsky's linguistic paper.
        • hansvm 1y ago
          In some fields, sure, cite the 4chan source, ideally with an archived link. Pure math tends to be much more conservative in citations than other fields though, and even when writing a paper about a longstanding math problem you wouldn't necessarily bother to include existing solutions. You reference the things you actually used, and even then you assume some common background knowledge for your audience and don't reference every little undergrad topology theorem or whatever. The point is to be honest with the reader about what was helpful for this work in particular, both to properly attribute things you actually used and to make any searches based on your work more targeted and fruitful.
  • constantcrying 1y ago
    >It’s weird how random people can submit non peer reviewed articles to preprint repos. It is weird how people use a platform exactly how it is supposed to be used.
  • T-A 1y ago
    > It’s weird how random people can submit non peer reviewed articles to preprint repos. Assuming that you are referring to the Arxiv, they can't: https://info.arxiv.org/help/endorsement.html https://info.arxiv.org/help/endorsement.html
  • naveen99 1y ago
    I would start with github. Arxiv, show hn, twitter, TikTok , Super Bowl ad if you are going for max look at me, i am not wearing diapers effect.
• brudgers 1y ago
  Previous submission comments, https://news.ycombinator.com/item?id=32341770 https://news.ycombinator.com/item?id=32341770
• bmacho 1y ago
  Article is from 2012, compare [0] and [1]. The pdf got replaced for some reason (bug, sensitive information in the meta or idk), but the article seems to have stayed the same, except the date. [0]: https://arxiv.org/pdf/1201.2590v1.pdf https://arxiv.org/pdf/1201.2590v1.pdf [1]: https://web.archive.org/web/0if_/https://arxiv.org/pdf/1201.2590v1.pdf https://web.archive.org/web/0if_/https://arxiv.org/pdf/1201....
• robwwilliams 1y ago
  Yes old, but even worse, it is not a well argued review. Yes, Bayesian statistics are slowly gaining an upper hand at higher levels of statistics, but you know what should be taught to first year undergrads in science? Exploratory data analysis! One of the first books I voluntarily read in stats was Mosteller and Tukey’s gem: Data Analysis and Regression. A gem. Another great book is Judea Pearl’s Book of Why.
  • wiz21c 1y ago
    Definitely. It always amazes me that in many situations, I'm applying some stats algorithm just to conclude: let's look at these data some more...
  • nxobject 1y ago
    On the subject of prioritizing EDA: I need to look this up, but I recall in the 90s a social psychology journal briefly had a policy of "if you show us you're handling your data ethically, you can just show us a self-explanatory plot if you're conducting simple comparisons instead of NHST". That was after some early discussions about statistical reform in the 90s - Cohen's "The Earth is round (p < .05)" I think kick-started things off.
  • jononor 1y ago
    Yes. And the same for DS/ML people also, please. The amount of ML people that can meaningfully drill down and actually understand the data is surprisingly low sometimes. Even worse for being able to understand a phenomena _using data_.
    • Charon77 1y ago
      When you have a lot of fancy metrics/models/bootsraps to throw at, people would just see what sticks.
      • jononor 1y ago
        Happens all the time. Problems come quickly when the datasets used for evaluation are not clean, or the evaluation is incorrect - data leakage, problematic imbalance between groups, distribution shifts vs the actual production data. Or people just checking the average performance, but not typical or worst case. Have seen many people run in circles chasing metrics that are meaningless to the task they are supposed to be solving.
• perrygeo 1y ago
  Frequentists stats aren't wrong. It's just a special case that has been elevated to unreasonable standards. When the physical phenomenon in question is truly random, frequentist methods can be a convenient mathematical shortcut. But should we be teaching scientists the "shortcut"? Should we be forcing every publication to use these shortcuts? Statistic's role in the scientific reproducibility crisis says no.
  • kccqzy 1y ago
    Frequentism methods are strictly less general. For example Laplace used probability theory to estimate the mass of Saturn. But with a frequentist interpretation we have to imagine a large number of parallel universes where everything remains the same except for the mass of Saturn. That's overly prescriptive of what probability means. Whereas in Bayesian statistics what probability means is strictly more general. You can manipulate probabilities even without fully defining them (maximum entropy) subject to intuitive rules (sum rule, product rule, Bayes' theorem), and the results of such manipulation are still correct and useful.
    • deleted 1y ago
      [deleted]
    • perrygeo 1y ago
      Drawing a sample of Saturns from an infinite set of Saturns! It's completely absurd, but that's what you get when you take a mathematical tool for coin flips and apply it to larger scientific questions. I wonder if the generality of the Bayesian approach is what's prevented its wide adoption? Having a prescribed algorithm ready to plug in data is mighty convenient! Frequentism lowered the barrier and let anyone run stats, but more isn't necessarily a good thing.
      • IshKebab 1y ago
        I dunno about you guys but I have no problems imagining randomly sampling Saturn.
• hnuser123456 1y ago
  Okay, apparently this is the core of the debate?: Frequentists view probability as a long-run frequency, while Bayesians view it as a degree of belief. Frequentists treat parameters as fixed, while Bayesians treat them as random variables. Frequentists don't use prior information, while Bayesians do. Frequentists make inferences about parameters, while Bayesians make inferences about hypotheses. --- If we state the full nature of our experiment, what we controlled and what we didn't... how can it be a "degree of belief"? Sure, it's impossible to be 100% objective, but it is easy to add enough background info to your paper so people can understand the context of your experiment and why you got your results. "we found that at our college in this year, when you ask random students on the street this question, 40% say this, 30% say this..." and then considering how the college campus sample might not fully represent a desired larger sample population... what is different? you can confidently say something about the students you sampled, less so about the town as a whole, less so about the state as a whole... I don't know, I finished my science degree after 10 years and apparently have an even mix of these philosophies. Would love to learn more if someone's inclined.
  • usgroup 1y ago
    I think Bayesian methods have made ground in sciences such as Sociology, Psychology and Ecology, which are mostly observational, but still attempt to make models with intepretable parameters. With observational studies, representing confounders and uncertainty is a primary concern, because they are the most important source of defeater. Here, Bayesian software such as brms, Stan, pyMC, become a flexible way to integrate may sources of uncertainty. Although, I suspect methods like SEM still dominate for their use cases. Personally, I find myself using Bayesian methods in a similar bag-of-tricks way that I use Frequentist methods mostly because its difficult to believe that complex phenomena is well described by either, so I use whatever makes the case best.
  • joshjob42 1y ago
    Well even for simple things there's a large difference. Say you toss a coin N times and observe heads x times. What is the probability of your next toss coming up heads? A frequentist arguably would say the question doesn't really have any meaning since probabilities are about long run frequencies of things occuring. They might do various tests or tell you the probability of that outcome under various probabilities for heads. A Bayesian would make an initial assumption about the probability of any given probability, and then compute a posterior using the likelihood function the frequentist may have, and give you a distribution for what you should believe about the what the true probability of heads is on your next coin toss. In general, the latter is more meaningful and informative. There's also pretty good arguments that any coherent method of representing credences is isomorphic to probability, see Cox's theorem.
• deleted 1y ago
  [deleted]
• throwaway81523 1y ago
  Some time back I remember a blog post about stuff you could straightforwardly do with frequency statistics that were much more difficult with Bayesian methods. I thought I bookmarked it but have no idea where it is now. I half remembered it being on Andrew Gelman's blog, but I spent a while looking there for it. No luck.
• usgroup 1y ago
  I consider myself an applied Statistician amongst other things, and I find this to be an ideological take mostly. When we do Statistics, we are firstly doing Applied Mathematics, which we are secondly extending to account for uncertainty for our particular problem. Whether your final model is good will largely depend on how it serves the task it was built for and/or how likely its critics believe it is to be falsified in its alternative hypothesis space. That is, a particular uncertainty extension is not necessary nor sufficient. For less usual examples, engineers may use Interval Arithmetic to deal with propagation uncertainty, quants might use maximin to hedge a portfolio, management science makes use of scenario analysis (deterministic models under different scenarios): all deal with uncertainty, none necessarily invoke either Frequentist or Bayesian intuitions. So, in my opinion, the most useful thing to teach neophytes is how to model with Maths. Second, it is how to make cases for the model under uncertainty.
  • derbOac 1y ago
    I've published stats papers on Bayesian methodology, consider myself a kind of Bayesianist, and stances like the link essay, and many of the posts here, make me sad and a bit frustrated. As you're saying, it's become an ideological, not functional, debate. Different inferential paradigms have their benefits and costs, and Bayesian methods have their own problems. Whenever you make an inference, it's a gamble, in a sort of literal sense. The choice of frequentist versus Bayesian methods basically is a bet about whether the bias due to your prior is small enough to offset the variance due to the lack of one. If the bias due to a "misguided" prior is high enough, you'll end up making a worse inference than not using a prior. You can, of course, use an optimally conservative prior, a reference prior, but for many cases what you're left with is a uniform prior, and therefore, frequentist inference. It's not always the same as a uniform prior, but in practice is often the same. I think the philosophical arguments against frequentist lines of thoughts are often unresolvable and involve strawmen characterizations of frequentism often derived from misperceptions of young adults learning any sorts of stats for the first time. There's also a very strong argument against strong priors in competitive situations, such as in achievement tests for admissions or some such thing. Imagine making priors for your test score based on your demographic background, for instance. Technically this might be ok from a subjective Bayesian perspective but I think almost no one would agree this is acceptable, and the reasons why are telling about statistical inference more generally. Many of the other arguments equally apply to Bayesian methodology too. The worst problems of NHST are not actually about the tests, they're about how they are used and interpreted, and would creep up if everyone was using credibility intervals too. I kind of think Bayesian methodology should be taught more at an earlier stage, but it would be irresponsible in my mind to not teach frequentist methodology at the same time.
• getnormality 1y ago
  What I hear when I read this: the way we do things today has definite and well-known problems. Wouldn't it be wonderful to do things in a different way whose problems are not yet well-understood or widely known?
• nurettin 1y ago
  You have a jar with five green, three blue marbles. First random marble you pick is green, what is the chance you get blue next? There is no use for bayes here. Now you have two jars you can't see inside of, 5g/3b and 3b/5r. You take one jar and want to guess which one you picked. You start pulling marbles and updating your priors until you reach an acceptable certainty. Now you have to use bayes or similar. These are tools, not ideologies. People who pit these tools against each other are demagogues.
  • NeutralCrane 1y ago
    > You have a jar with five green, three blue marbles. First random marble you pick is green, what is the chance you get blue next? There is no use for bayes here. There’s no use for Frequentism there either, it’s basic probability theory.
    • nurettin 1y ago
      And "frequentism" uses that basic probability theory, unless you have a "frequentist" answer to the first example that is something other than 3/7?
      • ratorx 1y ago
        Are you implying that Bayesian doesn’t use basic probability theory or gives a different answer? To calculate the simplest possible conditional probability, you need to have defined probability first.
        • nurettin 1y ago
          The first example is not a demonstration for bayesian inference. It uses a simple tool used in the "frequentist" approach. The bayesian approach example is below that. My point is the two are mutually exclusive in these scenarios. I am trying to demonstrate that there are formulas for different scenarios and you don't pit them against each other. The response to that was just pedantic.
• chuckadams 1y ago
  This helped a non-statistician like myself understand what it is I was supposedly taught wrong: https://xkcd.com/1132/ https://xkcd.com/1132/ (I still only sorta get it: I know it's reductionist so as to be funny, but to me those dice are quite literally a hidden variable)
  • firejake308 1y ago
    You're right that the dice are literally intended to represent a hidden variable. The difference between the frequentost and the Bayesian is that the frequentist only accounts for the hidden variable, whereas the Bayesian also accounts for the pre-test probability of the sun exploding. Since the probability of the sun exploding is 1 in a bazillion, multiplying that by 35 still gives you a very very low post-test probability that the sun has gone nova because 35 in a bazillion is still pretty unlikely
  • usgroup 1y ago
    It depends what the null hypothesis is here, but by construction, under a reasonable null, the p-value for an appropriate test would not be acceptable under a Frequentist framework.
  • rawgabbit 1y ago
    It is like running a casino versus gambling as an individual. If I was running a casino and presiding over tens of thousands of bets daily, I would use frequency statistics to guarantee the house always wins. If I was an individual gambler and determining my odds at a particular bet, I would use Bayesian.