8 ms·
Simpson's paradox
- roenxi 3y agohttps://en.wikipedia.org/wiki/Berkson%27s_paradox https://en.wikipedia.org/wiki/Berkson%27s_paradox is also one to be aware of. There are lots of ways for error to creep in when populations are created in a biased way. These two effects explain a lot of the stupid decisions that come out of "data driven" processes. It is common for data to suggest the opposite of the truth.
- pdonis 3y ago> It is common for data to suggest the opposite of the truth. Actually, I think the best takeaway from phenomena like these is that just doing statistics on a set of data can't tell you "the truth". If you don't understand the actual causal factors in play, your knowledge is very limited, no matter how much data you have or how many different ways you slice the statistics. For example, in the UC Berkeley case described in the Simpson's Paradox article, the data actually doesn't tell you anything useful about "bias" in the sense of "something people are doing that they should do differently to make the admissions process fairer". It doesn't even tell you where to look for possible "bias" without knowing more about the admissions process: it is controlled primarily by departments or by the university as a whole?
- staunton 3y ago> just doing statistics on a set of data can't tell you "the truth". If you don't understand the actual causal factors in play, your knowledge is very limited I would argue that ultimately, all your knowledge and understanding comes from "doing statistics on data". Maybe the statistics is done by sloppy slurpy things in the brain instead of in R, and maybe it's actually mathematically unsound most of the time, but it's still some sort of statistics.
- yccs27 3y agoI think the key difference is between statistics on passively collected data vs results from active experiments. The former will only ever show correlations, while the latter can prove causal results from the actions of the experimenter.
- pdonis 3y agoAlso, results from active experiments aren't limited to statistics. You can set up experiments to have discrete results, where no statistics is required to test a hypothesis. For example, the GHZ experiment [1] can rule out local hidden variable models and confirm QM predictions with no statistics at all: the two different models make contradictory predictions with no continuous variation between them. [1] https://en.wikipedia.org/wiki/GHZ_experiment https://en.wikipedia.org/wiki/GHZ_experiment
- staunton 3y agoIf you read a good experimental paper on GHZ, you will find quite a bit of statistics.
- pdonis 3y agoSure, but that doesn't contradict what I said. From the Wikipedia article I referenced: "For specific combinations of orientations, perfect (rather than statistical) correlations between the three polarizations are predicted by both local hidden variable theory (aka "local realism") and by quantum mechanical theory, and the predictions may be contradictory." "Perfect" correlations means, as the parenthetical comment shows, "doesn't require statistics to check".
- staunton 3y agoYou said > the GHZ experiment [1] can rule out local hidden variable models and confirm QM predictions with no statistics at all However, one needs to use statistics to even show GHZ works. That does sound contradictory to me. The correlations you get in experiments are never perfect and in this case they can be pretty far from perfect.
- andirk 3y agoIs the UC Berkeley case a good example of the importance of normalizing data before analyzing? Where things need to be put on a level playing field and handicaps applied to remove auxiliary noise.
- pdonis 3y agoNormalizing data doesn't fix the issue in the UC Berkeley case, because you still have to pick what to normalize over: do you normalize over the entire university, or separately over each department? The answer to questions like that can't be found in the data. You have to go look at how the university admission process actually works, and what roles the university vs. the individual departments play in it.
- o11c 3y agoOf course, an empiricist can just point to https://en.wikipedia.org/wiki/Goodhart%27s_law https://en.wikipedia.org/wiki/Goodhart%27s_law
- tunesmith 3y ago> Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes. It may be true in some cases, but I think the "Any" is disproven by the existence of proper scoring rules.
- o11c 3y agoIt's a very big assumption that such rules can, let alone do, exist in the first place.
- JacobAldridge 3y agoAs the aphorism goes, there are Lies, Damned Lies, and Statistics.
- r_thambapillai 3y agoI once encountered this in the real world as a data analyst a long time ago. I was working at an e-commerce company, called The Hut Group, and the whole year our marketing team had been saying our marketing cost of goods sold (the percentage of our revenue we needed to spend on marketing) had been declining across every product category. But at year end, the execs were shocked to realize that our cost of goods sold had almost doubled, from 10% to nearly 20%. The finance team had asked me to double check the marketing team's numbers, to see if there'd been some funny math in the reporting. But the marketing team were totally right, marketing spend across the three main categories - games, beauty, and nutrition had all fallen (~15% to ~10%, ~30% to ~25%, and ~50% to ~30% respectively). However, the mix of these product categories had shifted massively, with nutrition growing from roughly 10% of our total sales to now nearly 50%. In net that meant that whilst the marketing team had gotten more cost-efficient at selling every individual product category, the growth in the nutrition industry had vastly outstripped the growth in all other categories, and since that was the highest individual category, the aggregate marketing costs % had gone up, even though the team had improved every category. I then had the fun job of explaining the Yule Simpson paradox to a bunch of accountants.
- throwaway98797 3y agoit’s shocking that product mix wasn’t slide on reporting but marketing selects for positivity not objectivity the facts and only the facts that support what they do
- Anon84 3y agoIt’s actually surprisingly common. You can even find it in “classical” toy datasets like Iris: https://github.com/DataForScience/Causality/blob/master/1.2%20-%20Simpsons%20Paradox.ipynb https://github.com/DataForScience/Causality/blob/master/1.2%...
- appplication 3y agoCovid vaccination rates and deaths were rather famously subject to it. E.g. some combination of stats like “most covid deaths were vaccinated individuals”, “vaccination reduces death rate”, and “population segment with lowest vaccination rates has lowest covid death rates.” were all true at the same time.
- bee_rider 3y agoFor all of the examples on Wikipedia, it seems like there was some confounding extra variable that was missed. I wonder if anybody knows of a case where it just sort of happened randomly, with no big underlying cause? Or maybe I’m thinking of it wrong and this is impossible.
- eadler 3y agoIt can happen any time there is a mix shift in the underlying quantity of the subgroups. It's just that random changes in quantities are not likely to be studied or reported. It's easy to generate manually though.
- DonsDiscountGas 3y agoIn order for it to count as Simpsons paradox I think there would need to be a confounding variable. It's certainly possible for it to appear spuriously, and for something to look like a confounder when it isn't, but there would need to be some type of subgroup.
- Kalium 3y agoThe classic example of admissions does not have a missing confounding variable. It's a case of aggregation at the wrong level.
- waldrews 3y agoPrevious discussion: https://news.ycombinator.com/item?id=791821 https://news.ycombinator.com/item?id=791821
- esafak 3y agoI did not know Simpson's paradox was an object lesson in causal inference until the other day. The right paradigm dispels the paradox. Here's a better article: https://plato.stanford.edu/entries/paradox-simpson/ https://plato.stanford.edu/entries/paradox-simpson/
- deleted 3y ago[deleted]
- kromem 3y agoI absolutely love the Ellenberg quote: > Mathematician Jordan Ellenberg argues that Simpson's paradox is misnamed as "there's no contradiction involved, just two different ways to think about the same data" and suggests that its lesson "isn't really to tell us which viewpoint to take but to insist that we keep both the parts and the whole in mind at once." Keeping multiple possibilities in mind at once was what allowed the Epicureans to determine survival of the fittest, trait inheritance from each parent, that light was made of discrete units that weighed very little and were moving very fast, and that in order for free will to exist the quanta making up matter had to have multiple possible results under the same governing physical laws and conditions - all several millennia before the scientific method independently found the same results. It's a great analytical method, especially in data analysis as suggested here.
- biomcgary 3y agoInteresting list of Epicurean theories. Any good starting point that addresses them together and how multiple possibility thinking is related?
- kromem 3y agoI can't recommend enough straight up reading Lucretius'sNature of Things. But one of the examples in there of how their methodology ends up successful is when he's discussing the possible reasons lighting and thunder occur at different times. One possibility thrown out is that they are actually occurring at different times. But another is that they occur at the same time but one takes longer to reach the viewer than the other. On its own, these two ideas don't indicate the correct answer. But then Lucretius ties the latter to another observation - that this seems similar to how a drummer in the distance can be seen to beat the drums before you would hear the drums. Essentially in an age without the methodology of testable predictions, they circumvented that shortcoming by considering multiple hypotheses for multiple naturally occurring observations and looking for overlaps between them. This seems to have pointed them in the correct direction on a number of major topics, especially relative to their contemporaries who were generally arguing for a particular hypothesis with various appeals to rhetoric or principle (like Aristotle claiming the leader of a bee hive couldn't be female because it had a stinger and "the gods don't give women weapons"). The times the Epicureans completely miss the mark is generally when they disregarded their principle of avoiding false negatives and discounted things with insufficient observational evidence (for example, they had pretty bad cosmology and they rejected the Stoic pre-gravity due to their incorrect base assumption of infinite amounts of matter). The times they kept an open mind and considered how concepts overlapped, even when they were wrong about the 'why' of an initial assumption they were often correct in secondary assumptions when tying it into multiple other systems and observations.
- CrazyStat 3y agoWhen I taught intro stats many years ago I used to use house prices as a nice example of Simpson's Paradox (with actual data, for the students to investigate as part of a computational lab). The data I had was on US house sales from 2008, so it's 15 years out of date now--perhaps things have changed since. At the time, the average price for single-family house sales was higher for houses without central AC than for houses with central AC. Yet when you split the data down by state, in every state the relationship was reversed: houses with central AC were more expensive than houses without. The higher nationwide average price of houses without central AC was driven primarily by the large number of expensive houses in California without central AC.
- bruce511 3y agoIm reading this outcome to be the reverse of the examples above. Or perhaps identifying the correct stat to use based on your goals. In other words, in this case I don't really care what the national average is. I care about my house, my street, my area. In other cases, like in marketing, the stat that matters first is overall net profit. From there we can burrow down to understand the factors. In which case we come across business share before marketing spend. In the networking example, the goal is usage (throughput). Not speed or latency. Drawing the wrong stat first leads to incorrect conclusions.
- Gibbon1 3y ago> like in marketing, the stat that matters first is overall net profit. I have a take on that. The stat that matters is the profit per unit of non scalable business resource. As in how much management, marketing, sales, accounting, and engineering time does the product take per unit. It's important because those are often hard to scale. You can have a low margin product that requires zip of the above and it's good business. And the reverse, high margins but requires too much of the above and it's bad.
- CrazyStat 3y agoRight, one of the interesting things about Simpson's paradox is that there's not a uniform right answer: sometimes you care about the overall average, sometimes you care about the averages of subpopulations. You have to judge that based on the situation. One of the other comments linked [1] which includes Judea Pearl's analysis of Simpson's paradox from a causal inference point of view [2], which lays this out nicely (though maybe not easy to understand--it took me many hours of study to get comfortable with Pearl's causal inference work, even with a strong stats background). [1] https://plato.stanford.edu/entries/paradox-simpson/ https://plato.stanford.edu/entries/paradox-simpson/ [2] https://plato.stanford.edu/entries/paradox-simpson/#ConfPearAnalPara https://plato.stanford.edu/entries/paradox-simpson/#ConfPear...
- paradocks 3y agoLord's paradox is closely related to Simpson's, and I find it's a little easier to understand visually: https://repository-images.githubusercontent.com/597130499/46354e44-b04b-454c-87a5-4fc24036f7f1 https://repository-images.githubusercontent.com/597130499/46... Imagine the horizontal axis is dose of a drug, and the vertical is the response, like hours of sleep. Looking at Lisa's response, it's clear that increasing the dose reduces sleep. Same for Bart. But if you do a linear regression of all the data, shown by the red line, dose increases sleep, which is wrong.
- kfarr 3y agoUpon first glance I assumed it was the Simpson's episode about Mr. Burns having "a vast range of diseases so great in fact that they cancel each other out," explained here: https://simpsons.fandom.com/wiki/Three_Stooges_Syndrome https://simpsons.fandom.com/wiki/Three_Stooges_Syndrome Surprised at the similarity but of course that was probably on purpose by the genius Simpsons writers of the late 90's.
- 1970-01-01 3y ago"A man can have as many diseases as he damn well pleases." https://en.wikipedia.org/wiki/Hickam's_dictum https://en.wikipedia.org/wiki/Hickam's_dictum
- xcdzvyn 3y agoI assumed it'd be something about the number of recent significant events that the Simpsons "predicted"[0]. [0] https://www.independent.co.uk/arts-entertainment/tv/news/the-simpsons-predictions-came-true-b2403112.html https://www.independent.co.uk/arts-entertainment/tv/news/the...
- caditinpiscinam 3y agoMy gripe with most takes on the affirmative action debate is that they completely ignore this issue (in spite of the Berkeley case-study). It's trivial to take a set of admissions data and partition it by race to reveal a "racial bias", which would shrink or disappear if other factors correlated with race (like income) were accounted for.
- marklubi 3y agoWhat is the deal with posting (seemingly) random links to Wikipedia articles lately without context? Some of them are interesting, but most(?) of them come without details. Please provide me with some context.
- lukas099 3y agoThings hackers would find interesting. I think they have always been posted as long as I've been here.
- marklubi 3y agoThere is no context provided with any of these links. I'm not making any claims about whether or not they have interesting content. There's been a huge uptick in submissions of random Wikipedia articles where there is no context provided with them. There are maybe two or three in the last day that have anything beyond the title included. https://news.ycombinator.com/from?site=wikipedia.org https://news.ycombinator.com/from?site=wikipedia.org
- BeetleB 3y ago> There is no context provided with any of these links. Strange comment. There doesn't need to be.
- estiaan 3y agoI was reading the example of UC Berkely appearing to have gender bias in the admissions and read the following: “it showed that women tended to apply to more competitive departments with lower rates of admission, even among qualified applicants (such as in the English department), whereas men tended to apply to less competitive departments with higher rates of admission (such as in the engineering department)” That’s the opposite of what I would expect, I’d expect that English and the arts in general would be a lot easier to get into than stem, that’s how it is in Australia Edit: When I say get into I mean get into university, not getting into the industry
- tacitusarc 3y agoCompetitiveness is one measure of difficulty, but there are others. Engineering departments tend to be qualitatively more difficult to get into than the humanities, which are quantitatively more difficult.
- aldonius 3y agoI think it could be consistent. In Australia we usually think about undergrad competitiveness in terms of the minimum ATAR rank of last years' admissions, right? But you could also think about competitiveness in terms of admission fraction. And I think those two metrics can be consistent if a very popular degree has a disproportionately low fraction of high-ATAR applications.
- DonsDiscountGas 3y agoI was surprised to read that too. I think the answer is that here we're looking at admissions rate = number admitted / number applied, which is not the same as overall difficulty in a conceptual sense. The only people applying to grad school in math are people who got a BS in math and did so with good grades (or perhaps some other STEM field + significant theoretical math coursework). On the arts side I suspect they draw from a larger pool (plus people tend to switch from STEM to something else a lot more than the other way around) of backgrounds. It's easier to convince oneself that a short story is great (when others may disagree) than convincing oneself a math proof is correct when it objectively is not. So there's less self-selection on the applicant side, and hence a lower admissions rate.
- ninetyninenine 3y agoThat visualization was so effective I didn't need to read the wiki at all. I get it just from that.
- pcwelder 3y agoEncountered it recently. I had two different dataset to evaluate model performance on from different domains. One dataset was closer to training data and the other was closer to our business use case. The hypothesis was that performance on the latter dataset would be poorer due to overfitting. Indeed the accuracy on all categories had reduced. However, overall accuracy was much higher! This was because the second dataset had higher frequency of easy to predict categories. If we had just looked at overall number we would have concluded that there was no overfitting to train domain, which was not the case.
- EGreg 3y agoDoes this also explain the gender pay gap? Or is it only for trends?
- nomilk 3y agoThe short animation on the wiki page is a great example of a picture being worth 1000 words: https://en.wikipedia.org/wiki/File:Simpsons_paradox_-_animation.gif https://en.wikipedia.org/wiki/File:Simpsons_paradox_-_animat...
- freedomben 3y agoHot damn, you're not kidding. I was struggling a bit with it from reading the article text, but that animation clarified it for me in seconds!
- thefringthing 3y agoHere it is in one static image: https://i.imgur.com/qJsTjpp.jpeg https://i.imgur.com/qJsTjpp.jpeg
- LinAGKar 3y agoThat second graph reminds me of Shepard tones [1], known e.g. from the Super Mario 64 staircase, where each component is steadily rising in pitch, yet the tone as a whole stays exactly the same in the long term. [1] https://en.wikipedia.org/wiki/Shepard_tone https://en.wikipedia.org/wiki/Shepard_tone
- Alquds2024 3y ago[dead]
- topologie 3y agoAnd to think this is the "level 1" type of paradox in statistics and probability... It's a fun area.