6 ms·
Generating Simpson's Paradox with Z3
- incognito124 2y agoI just love the napkin equation in the middle of [1], it really made it clear to me [1]: https://robertheaton.com/2019/02/24/making-peace-with-simpsons-paradox/ https://robertheaton.com/2019/02/24/making-peace-with-simpso...
- joe_the_user 2y agoThe wikipedia gives a reasonable take: "Mathematician Jordan Ellenberg argues that Simpson's paradox is misnamed as 'there's no contradiction involved, just two different ways to think about the same data' and suggests that its lesson 'isn't really to tell us which viewpoint to take but to insist that we keep both the parts and the whole in mind at once.'" My own take is that any statistic has a value and a strength (in the case of averages, strength can be the number of instances averaged, for instance). You can have to keep in mind both. https://en.wikipedia.org/wiki/Simpson%27s_paradox https://en.wikipedia.org/wiki/Simpson%27s_paradox
- deleted 2y ago[deleted]
- kgwgk 2y ago> its lesson 'isn't really to tell us which viewpoint to take When the analysis is intended to inform a decision it’s important to take the appropriate viewpoint. https://ftp.cs.ucla.edu/pub/stat_ser/r414.pdf https://ftp.cs.ucla.edu/pub/stat_ser/r414.pdf https://www.unige.ch/cisa/files/9616/7501/5107/CISA_BM_statsupport_20230130_simpson.pdf https://www.unige.ch/cisa/files/9616/7501/5107/CISA_BM_stats...
- azinman2 2y ago[flagged]
- TheMrZZ 2y agoBiggest trap of Simpson's paradox is the results can change with every level of granularity. If you take the example of Treatment A vs Treatment B for tumors, you can get infinite layers of seemingly contradicting statemens: - Overall, Treatment A has better average results - But if you add tumor size, Treatment B is always better - But if you add gender to size, Treatment B is always better - But if you add age category to gender and size, Treatment A is always better - etc... It totally contradicts our instincts, and shows statistics can be profoundly misleading (intentionally or not).
- Narhem 2y agoLike Tumors are higher dimensional objects than what human brains are trained to perceive.
- oh_my_goodness 2y agoIf the previous comment is right, then this one is plainly also true in some sense. I'm disappointed to see downvotes.
- TheMrZZ 2y ago> If the previous comment is right I actually coded a Z3 program to prove it! The 3-variables version takes too long to resolve, but I got results for the 2-variables version (tumor size + gender): Results can be found in this GSheet: https://docs.google.com/spreadsheets/d/1tsBhElTgXjVTeas8quarGE7Go4n9DZ1aTUomnvUeIKs/edit?usp=sharing https://docs.google.com/spreadsheets/d/1tsBhElTgXjVTeas8quar... Code is here: https://gist.github.com/TheMrZZ/c33927ca2cc917997a67d7f84b8205c9 https://gist.github.com/TheMrZZ/c33927ca2cc917997a67d7f84b82... I'm currently running the 3-variables version, hopefully I'll get results this afternoon. We can clearly see the same problems that arise in the 1-variable Simpson's paradox (widely different population sizes).
- gradschoolfail 2y agoFor pedagogues and practitioners alike: there is a subtle connection between Simpson’s paradox and the wild geometry of relative entropy. This might be partly why effect sizes are also contentious. Besides Ellenberg’s mind-altering discussion of that link[1], see hints on the second page of: https://www.qeios.com/read/XB1N2A/pdf https://www.qeios.com/read/XB1N2A/pdf [1] "[the point of Simpson’s paradox] isn't really to tell us which viewpoint to take but to insist that we keep both the parts and the whole in mind at once." Ellenberg, from Shape: The Hidden Geometry of Information, Biology, Strategy, Democracy, and Everything Else (2021)
- tombert 2y agoZ3 is kind of my new favorite thing right now. I have a problem that lends itself quite well to constraints-based reasoning, and I need it to be optimized. I'm sure I could have hacked something together using any number of programming languages, but after playing with Z3 for a bit, I realized that this could be easily done in around ~100 lines of an SMT2 file, and probably be considerably faster. Tools like this make me feel a lot better about all the time I wasted playing with predicate logic.
- samstave 2y agoCurious, would your data constraints be related to http://equationofhappiness.com/ http://equationofhappiness.com/ Would like to know how you us Z3 to evaluate emotional happy Q. Could you apply that to evaluate a piece of content (like thread, feed, comment) and then evaluate the energy of the thing for its happiness quotient value whatever? then you can just Z3 the thread and determine the psychological predicted impact... Maybe you could then design casula games that provide the positive happy Q vibes. -- I was in a program from when I was a baby with UCSD that was a life tracking project and they would check in with you every so often to see where you were on that trajectory - and where you were happy in life etc. Problem is that it was also tied to Morehouse University, MK, Vic Baranco, and a bunch of other Stanford thingy's from the 70s that we all know abou these days.
- tombert 2y agoNah, the core math for Equation of Happiness is my dad's thing, and it's using a genetic algorithm for its optimization stuff, all I did was port some of his code to Julia and write a basic web frontend. If I had known about Z3's optimization tools when he was writing the book, I might have tried to use it, but I'm not sure how well Z3 would actually work with the differential equation stuff he was doing, since it's not really discrete. I'm working on something that is trying to utilize Z3 for some financial market stuff that I hack on in my free time.
- darkteflon 2y agoSame. I’d been circling SMT solvers for a while but a recent HN post on knuckledragger[1] (built on Z3) made me finally take a closer look at Z3 itself. It kind of feels like stealing fire. [1] https://news.ycombinator.com/item?id=41161455 https://news.ycombinator.com/item?id=41161455
- clircle 2y agoSeems like an inappropriate use of the word "paradox". How about Simpson's intuitive situation?
- travisjungroth 2y ago“Going against common expectation” is well established as one of the definitions of “paradox”. Check any dictionary you choose.
- reallybees 2y agoParadoxical iff data sets are isolated; explains the Gemini effect. Preferences tend to model future outcomes if sample sizes of surveyors are combined.
- eadler 2y agoIt is a veridical paradox, not a falsidical paradox. A falsidical paradox is what most people think of as a formal paradox: from the assertions you derive a conclusion which is false (either deductively or inductively). A veridical paradox is where you derive a true result which appears absurd. it is an informal classification based on intuition. https://en.wikipedia.org/wiki/Banach%E2%80%93Tarski_paradox https://en.wikipedia.org/wiki/Banach%E2%80%93Tarski_paradox is a classic veridical paradox.
- staplung 2y agoSimpson's Paradox keeps experimenters up at night because it embodies the idea that although your data might say one thing, it's always possible that slicing the data via some unknown axis of finer granularity might paint a very different picture. It's hard to know if there is such an axis lurking there in your data, let alone, what it might be. If you get paranoid about its presence it can lead you to second guess pretty much every statistic. "I know that 4 out of 5 dentists recommend chewing X Brand gum but what if I slice the dentists by number of eyes? Maybe both one-eyed dentists and two-eyed dentists aren't so enthusiastic."
- grobbyy 2y agoIt does, but only experiments who don't have a strong grounding in research methods. Simpson's paradox is part of a broader problem: correlation does not imply causation. In practice, it's one of many problems with making decisions based on correlations. In a randomized control trial, with a large random sample, the odds of Simpson's Paradox coming up are astronomically low. Good statisticians WILL second-guess ANY conclusions based on post-hoc data analysis. To frame this in scientific jargon, exploratory data analysis and correlations are great at generating hypotheses, but those need to be confirmed with methods appropriate for confirmatory analysis.
- zekrioca 2y agoI dislike the baseball example, because it is too context specific to the other several Bi people who don’t follow the sport.
- 0xDEADFED5 2y agoDoes Bi = Baseball ignorant? I'm definitely Bi
- staplung 2y agoI know essentially nothing about Z3 but it seems like there's a potential problem in the code. There's a section headed by the comment "All hits and miss counts must be positive" followed by a bunch of assertions that those numbers are greater than 0. Isn't it possible that you have 0 hits or misses? I mean, what if your season is short and you only face a couple of lefty pitchers and strike out both times? In any case, I would explain the paradox differently than the author. The author says: "The key to understanding the paradox is that the players did not bat against the same set of pitchers. A batted against 5 lefties and 12 righties; B against 2 and 11." I would say instead that the key to understanding the paradox is to observe that both players are much better when batting against lefties and that player A batted against lefties much more often, both in absolute and relative terms. In other words, A is not as good against lefties as B but he faced a lot more of these comparatively easy pitchers.
- cortesoft 2y agoThat would be possible for a hitter, but it would be impossible to be an example of Simpson's paradox in that case.
- Izkata 2y agoThis visualization on Wikipedia was what I needed to understand Simpson's Paradox, the descriptions never made a whole lot of sense to me until seeing it like this: https://en.wikipedia.org/wiki/Simpson%27s_paradox#/media/File:Simpsons_paradox_-_animation.gif https://en.wikipedia.org/wiki/Simpson%27s_paradox#/media/Fil... Along the same lines of "visualize your data to see what's really going on" is Anscombe's Quartet: https://en.wikipedia.org/w/index.php?title=Anscombe%27s_quartet https://en.wikipedia.org/w/index.php?title=Anscombe%27s_quar... And then there's the Datasaurus [Dozen], which has some fun with the idea behind Anscombe's Quartet: https://en.wikipedia.org/wiki/Datasaurus_dozen https://en.wikipedia.org/wiki/Datasaurus_dozen (you can see it animated here: https://blog.revolutionanalytics.com/2017/05/the-datasaurus-dozen.html https://blog.revolutionanalytics.com/2017/05/the-datasaurus-... )
- mjburgess 2y agoA more general lesson is that correlation and causation are unrelated: the former doesnt imply the latter, and the latter does not imply the former. Just because one thing causes another does not mean it will be correlated with it. There is no contradiction in subsets having different correlations that the parent set. The apparent "paradox" arises from reading the data causally. The purpose of this lesson is to expose these assumptions in interpretation of data. Few seem to get the message though.
- weinzierl 2y agoIf anyone has doubts about the second claim, think about a hash function. The input certainly causes the output, but they are not correlated in a statistical sense.
- mjburgess 2y agoConsider a medicine which kills everything with kidneys. It perfectly correlates with killing everything with a liver. Consider another medicine which kills everything with kidneys, unless they have a liver (eg., which filters it). Now there is no correlation at all with an effect on the kidneys, nor will there ever be (since all animals with one have the other) unless someone deliberately impairs a liver.
- hackandthink 2y ago"The key to understanding the paradox is that the players did not bat against the same set of pitchers." This is misleading. I am baseball ignorant but I feel this is a contrived and bad example for Simpson's paradox. (UC Berkeley gender bias example is much better)
- kgwgk 2y agoThere you have something like "The key to understanding the paradox is that the schools did not have the same set of applicants.” It seems similar.
- hackandthink 2y agoMy take: The key to understanding the paradox is that women apply mostly to popular departments therefore women are declined more often than man who apply more to unpopular departments.
- kgwgk 2y agoOk, women and men applying to a different mix of easy/hard departments - not so different from the two players batting against a different mix of easy/hard pitchers.
- hackandthink 2y agoYes, easy/hard pitchers works better than right-handed/left-handed pitchers. Don't know anything about baseball, I would have assumed teammates battle the same pitchers.
- kqr 2y agoPopular wisdom regarding experimentation has always been to "vary just one thing at a time, keeping the others as constant as possible". Fisher argued to the contrary, that we should (systematically) try as many variations as possible simultaneously. Simpson's paradox (and perhaps the similarly counter-intuitive Berkson's paradox) are the reason why: when analysing just one variate at a time, we risk seeing relationships that aren't there, or run counter to what we are trying. Proper multifactor analysis that accounts for all variations simultaneously is required to learn about complex phenomena.
- cubefox 2y agoA basic question I always had about Simpson's paradox: If X is positively correlated with Y, but X is also negatively correlated with its parts when Y is broken down (Simpson's paradox) – is it then more likely that X causes Y or that X causes not-Y? This seems to be a pretty fundamental question but I have never seen it addressed.
- taeric 2y agoMy understanding is that the answer is more that there are more important/causal elements than X acting on Y. That is it. For the college admissions example, the more important factor was what department you applied to. That had far more of a meaningful contribution to whether you were admitted than what your sex was. So, if folks wanted to increase admissions, you wouldn't focus on sex, you would focus on expanding departments. That is the general trend with all of the examples. The paradox is you think focusing on X would be the important thing to focus on, but the data had hidden that there was a Z that you should instead focus on.
- im3w1l 2y agoWell it depends on context I would say. Like first you have to consider sample size. The more you break something down the lower the sample size becomes, and the more breakdowns you try, the more likely you will find some kind of spurious pattern. But let's say the effect is real. Then you have to start considering what is causing why, which is highly context dependant. To take the example of gender bias in the grad school admissions, men were more likely to be admitted but the effect reversed when breaking down by department. Hypotheses come to mind: A department being easy to get into causes a higher male ratio. (Subhypotheses: Maybe men want to take it easy? Maybe women seek out prestige?) High male ratio of applicants to a department causes it to become easier to get into. (Maybe funding flows to male departments?) There is some kind of unknown factor that both causes a high male ratio and a high rate of admissions (Maybe a booming sector attracts men and leads to easy admissions?) With all these possibilities I think it should become clear that there can't be a general solution to your question, you have to consider the context and dig deeper.
- cubefox 2y ago
- taeric 2y agoI am not sure I agree with that conclusion. In this particular example, it is almost certainly more important to consider frequency and sample size? Could have been the exact same pitchers, per se.