3 ms·
It's definitely arrogant. But after much experience trying to explain these things to people, I'm more and more convinced it's not just "if you just put it the
by kqr 2y ago
It's definitely arrogant. But after much experience trying to explain these things to people, I'm more and more convinced it's not just "if you just put it the right way they will understand". Sure, they will nod politely and pretend to understand, and may even do a passable job of reciting it, but once its cast in a slightly different light they are just as confused.
Much as with reading and writing, I think it takes an active imagination and a long slog of unlearning to trust logic (the "ought to" thinking that shields one from reality) and coming to terms with the race not being to the swift, etc, and that these effects can be quantified.
It's not that some people are incapable of it. Much like literal literacy has reached rates of 99.9 % in parts of the world, I'm convinced statistical literacy can too. But when your teacher is not statistically literate (which I hypothesise they are not, generally speaking), they will not pass that on to you. They will not give you examples where the race is not to the swift. They will not point out when things seem to happen within regular variation and when they seem to be due to assignable causes. They will not observe the battle against entropy in seating choices in the classroom. They will not point out potential confounders in propensity associations. They will not treat student performance as a sample from a hypothetical population. They will not grade multiple-choice questions on KL divergence, although that would be far more useful. I could go on but I think you get the point.
Yet to be clear, I'm not talking about just applying statistical techniques and tools. I'm talking about being able to follow a basic argument that rests on things like "yes I know they are a fantastic founder but startups fail 90 % of the time and so will they" or "if the ordinary variation between bus arrivals is 5–15 minutes and we have waited 20 minutes then there is something specific that put the bus we are waiting for into a different population." These are not obvious things to many people.
This is not a personal failure – it is a lack of role models and teachers. I wouldn't have considered myself statistically literate until recently, and only thanks to accidentally reading the right books. I wouldn't even have known what I was missing were it not for that!
I suspect it will take a few generations to really get it going.
If someone would donate me large amounts of money I would love to actively research this subject, come up with reliable and valid scales to measure statistical literacy, and so on. But in the meantime I can only think in my spare time and sometimes write about it online.
- shever73 2y agoWhat resources would you recommend for someone who wants to improve their statistical literacy? You mention reading the right books, I'd appreciate it if you could give a short list, if you have time.
- chuankl 2y agoI am not the person you asked, but I have been on a similar path to improve my statistical literacy. For context, I am fairly good at math generally (used to be a math major in college decades ago; didn't do particularly well though I did graduate) but always managed to get myself extremely confused when it comes to statistics. In terms of books: there are a few good ones aimed for the general public, such as The Signal and The Noise. How to Measure Anything: Finding the Value of Intangibles in Business is a good book of applying statistical thinking in a practical setting, though it wouldn't help you wrap your brain around things like the Monty Hall problem. The one book that really made things click for me was this: Probability Theory: The Logic of Science by E. T. Jaynes This book is a bit more math-heavy, but I think anyone with a working background in a science or engineering field (including software engineering) should be able to get the important fundamental idea out of the book. You don't need to completely comprehend all the details in math (I surely didn't); it is enough to have a high-level understanding of how the formulas are structured at the high level. But you do need enough math (for example, an intuitive understanding of logarithm) for the book to be useful.
- kqr 2y agoI second both The Signal and the Noise as well as How to Measure Anything. I also mentioned upthread Willful Ignorance. I think perhaps the best bang for your buck could be Wheeler's Understanding Variation -- but that is based mainly on vague memory and skimming the table of contents. I plan on writing a proper review of that book in the coming year to make certain it is what I remember it to be. I think the earlier works by Taleb also touch on this (Fooled by Randomness seems to have it in the title). But then I strongly recommend branching out to places where these fundamentals are used, to cement them: - Anything popular by Deming (e.g. The New Economics) - Anything less popular by Deming (e.g. Some Theory of Sampling) - Moneyball - Theory of Probability (de Finetti) - Causality (Pearl) - Applied Survival Analysis - Analysis of Extremal Events - Regression Modeling with Actuarial and Financial Applications The more theoretical and applied books are less casual reads, obviously. They also happen to be the directions in which I have gone -- you may have more luck picking your own direction for where to apply and practice your intuition. Edit: Oh and while I don't have a specific book recommendation because none of the ones I read I have good opinions on, something on combinatorics helps with getting a grasp on the general smell of entropy and simpler problems like Monty Hall.