8 ms·

Simpson's Paradox

210 points by mmaia 13y ago

sp332 13y ago
Reminds me of Anscombe's Quartet: four datasets that have the same mean, variance, correlation, and linear regression, but are really very different. http://en.wikipedia.org/wiki/Anscombe%27s_quartet http://en.wikipedia.org/wiki/Anscombe%27s_quartet
- lewis500 13y ago
  maybe we'll do that one next!
tel 13y ago
Simpson's Paradox is scary if you've not seen it before, but you should not stop here and instead proceed immediately on to omitted variable bias and the conversation here (http://normaldeviate.wordpress.com/2013/06/20/simpsons-paradox-explained/ http://normaldeviate.wordpress.com/2013/06/20/simpsons-parad...). In short, Simpson's paradox occurs because probability distributions and causal claims are distinct things which behave differently. It's nothing more than a particular, insidious example of correlation not implying causation. It's perfectly possible for a probability distribution to have a "contradictory" shape, but perfectly impossible for logical statements about the world to be contradictory. The resolution is that you shouldn't let your probability distributions turn into logical statements without analyzing your causal assumptions. This will lead you to whether or not excluding a variable is omitted variable bias (and whether including an improper one will lead to included variable bias, which is rarely recognized).
tlb 13y ago
The Omitted Variable Problem is part of my mental framework that causes me to not believe most epidemiological studies, especially ones that confirm a popular belief. The almost universally omitted variable is health-consciousness. Some people are health-conscious and some aren't. People who are health-conscious do a whole bunch of things, some of which help (like exercise, sleep well, eat moderately). They also do things that are widely believed to be good for you, like eating broccoli. So if you do a study, you'll find that people who eat lots of broccoli are healthier. You'll be able to confirm pretty much any widely believed health folk wisdom, unless it's something quite harmful, as long as you omit health-consciousness as a variable.
- maxerickson 13y ago
  If it was a huge deal, wouldn't epidemiologists be slobbering over the statistical power they would get from using it? That is, if they can control for it, they can make assertions about smaller effects.
  - gwern 13y ago
    > If it was a huge deal, wouldn't epidemiologists be slobbering over the statistical power they would get from using it? Why would you want to do that? You can build a nice career on publishing (spurious) associations. As long as relevant variables are omitted, no epidemiologist need ever be unemployed.
    - maxerickson 13y ago
      I'm optimistic enough to believe in a few wild cards trying to do good science. A whole bunch of the rest have to put a little bit of effort into competing with them. It occurred to me a while after my other comment that a belief in the ability of people that test as health conscious to actually do things that improve their health might not be especially justified. I mean, lots of people are quite happy with whatever quackery, maybe it really does all come out in the wash.
    - maxander 13y ago
      That may get you written about in popular-science magazines, but your work will have a low impact factor- this is largely measured in how many of your fellow researchers cite your work, and its crucial to an academic researcher's career. If your colleagues can easily see that your statistical analysis is flawed, they won't waste their own time doing work that builds off of yours.
      - wnewman 13y ago
        It is not necessarily true that just because they are called scientists, real people with real motives, funded by organizations of real people with real motives, operating in a political regime controlled by real people with real motives, will act that way. (By "that way" I mean "your work will have a low impact factor- this is largely measured in how many of your fellow researchers cite your work" and "If your colleagues can easily see that your statistical analysis is flawed, they won't waste their own time doing work that builds off of yours.") See e.g. Chan and Boliver "The Grandparents Effect in Social Mobility", http://asr.sagepub.com/content/78/4/662 http://asr.sagepub.com/content/78/4/662 (HT http://www.arnoldkling.com/blog/a-grandparent-effect/ http://www.arnoldkling.com/blog/a-grandparent-effect/) and many of the works that it cites. There seems to be a sizable mutually-citing "scientific" literature devoted to studying correlation in biological descent while dogmatically not controlling for ordinary DNA-based inheritance of psychological traits from parent to child. (Indeed, not even controlling for DNA-based inheritance of physical traits. Health and height and attractiveness are correlated with income, and are significantly physically heritable.) By not controlling for the properties which are transmitted by sperm meeting egg, your can find all sorts of fascinating possibilities to investigate to explain the correlation. (E.g. Chan and Boliver carefully note that "well-connected grandparents could also use their social contacts to help grandchildren with job searches." Good, good, carefully investigate all the possibilities.)
teamonkey 13y ago
The point of an interactive illustration like this is that it should make it more intuitive and easier to understand than the core concept, not more confusing. What is the meaning of the green and purple lines on the graph, why do they have different gradients and why can't I adjust them? Why does the Simpson's Paradox apply sometimes and not others? Why are there so many bars and donut charts? What does the gray circle around the donuts mean? Information overload.
- publicfig 13y ago
  I feel as thought both graphs with purple and green lines are accurately explained. Were you trying to skim through the article or were you still left confused after reading it? The top one is explained in the accompanying text and the bottom one is explained with a labeled x and y axis and relies on the information provided above.
  - teamonkey 13y ago
    If the article describes the subject adequately, what is the point of the graphic? I found the interactive graphic more confusing than the description. Compare and contrast to the clarity of their central limit theorem demonstation [1] vs its wikipedia page [2]. [1] http://blog.vctr.me/posts/central-limit-theorem.html http://blog.vctr.me/posts/central-limit-theorem.html [2] http://en.wikipedia.org/wiki/Central_limit_theorem http://en.wikipedia.org/wiki/Central_limit_theorem
    - publicfig 13y ago
      By that, I meant that the article described what the graph was representing, not that the article described the information presented adequately. Also, I absolutely disagree with the assumption you made that graphs shouldn't be present if the information is presented in the article. It's good to provide visual demonstrations when applicable in order to help clarify subjects.
- jere 13y ago
  It took a little playing around to figure out some of the graphics. The donuts indicate which group (men or women) was admitted more in each kind of department. Play around with the sliders and you can see it toggle in the combined row, while simultaneously a yes/no answer updates after the question "Simpson's paradox?" It would really help if yes/no was highlighted. Basically, you can make the paradox appear by dropping the % of women applying to easy departments. You can't make the donut switch in easy or hard departments individually... that's the whole point: women have a slight advantage in each. Honestly, I found the "Proper Pooling" section to have the most intuitive explanation. Combined, there is a bias. By department, there is no bias. The reason is clear: very few women applying to easier departments.
maxk42 13y ago
That is some sexy javascript.
- ZoF 13y ago
  My first thought as well
- vicapow 13y ago
  thank you! :D
trains 13y ago
It's nice visually but the text-based information could be more thorough - which would result in less links at the bottom.The more info contained on the page the better!
alexPetrov 13y ago
Reading up on Simpson's Paradox again made me realize something: women retrieving custody more often than men appears to me to be a perfect example of Simpson's Paradox. Overall, women get custody more often than men, this is true, but if you consider only the cases where men actually asked for or attempted to retrieve custody, this is no longer true[0]. [0]: http://www.villainouscompany.com/vcblog/archives/2012/04/child_supportcu.html http://www.villainouscompany.com/vcblog/archives/2012/04/chi...
- Tichy 13y ago
  How to interpret that, though? Seems likely to me that who asks for custody would be influenced by their chances of receiving it. How many of those cases are "contested custody"? Then that article places the claim "Additional evidence, however, indicates that women may be less able to afford the lawyers and experts needed in contested custody cases (see “Family Law Overview”) and that, in contested cases, different and stricter standards are applied to mothers." without providing any data. So it ends up being a propaganda piece, which makes it not very trustworthy.
dsego 13y ago
For me this was far better explained in the following text (also posted on HN not so long ago): http://www.michaelnielsen.org/ddi/if-correlation-doesnt-imply-causation-then-what-does/ http://www.michaelnielsen.org/ddi/if-correlation-doesnt-impl...
tensafefrogs 13y ago
Anyone interested in this stuff should definitely watch the lectures from the Stats 110 class from Harvard that's up on iTunes U (and perhaps other places). Lecture 6 talks about this paradox and if I recall correctly he might even talk about this exact case: https://itunes.apple.com/us/course/statistics-110-probability/id502492375 https://itunes.apple.com/us/course/statistics-110-probabilit...
jasonwatkinspdx 13y ago
Judea Pearl has formalized a resolution to Simpson's Paradox: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.34.8955&rep=rep1&type=pdf http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.34.... An oversimplification of his idea would be to say that given assumptions about the causal independence of variables, it becomes clear which way you should group the data. Although it's not always possible to make these independence assumptions, much of the time they are obvious and uncontroversial. Taking the Berkeley gender bias case as an example: We know it's possible that biological gender influences which department graduates apply to, but that it's impossible that the department a graduate applies to influences their biological gender. This fact alone tells us that we need to look at the data by department rather than in aggregate, resolving the paradox.
- gweinberg 13y ago
  Well, it certainly makes more sense to look at the by department data rather than the aggregate data. But I would say that the fact that we see such large disparities in which departments are applied to is already enough to refute the idea that male and female applicants are pretty much the same, and that therefore it would be pretty reckless to conclude any sex discrimination based on the difference in acceptance rates.
  - deleted 13y ago
    [deleted]
  - tel 13y ago
    And Pearl's point is that the "real world" logic you just applied is both important and not statistical—you need to augment your analysis with these causal assumptions in order to translate probability into meaningful causal statements.
- tel 13y ago
  That's a great reference. I'm going to use it from now on too.
  - 13y ago
cfontes 13y ago
Really interesting text, thanks. But I think this last statement "or Texas schools to waste money copying Wisconsin." was meant to be the other way around
- yen223 13y ago
  I may have read it wrong, but I think the statement is correct. Texas minorities outperformed Wisconsin's, but the stats made it look like Wisconsin's minorities did better.
  - penrod 13y ago
    To elaborate: Black students in Texas outperform black students in Wisconsin, hispanic students in Texas outperform hispanic students in Wisconsin, and white students in Texas outperform white students in Wisconsin. Overall test scores are higher in Wisconsin only because white students in both states are the highest scoring group, and Wisconsin schools have a higher proportion of white students than Texas schools. To the extend that any of this can be copied, it would probably be better for Wisconsin to improve the test scores of its ethnic groups to Texan levels, rather than for Texas to emulate Wisconsin's lilly-whiteness.
martimoose 13y ago
I had a hard time understanding the last graphic, because I was reading on an iPad, and the sliders don't work. I thought it was all static, and couldn't match the numbers in the graph with the previously mentioned numbers, as the defaults of the sliders do not give a simpson's paradox.
wunderlust 13y ago
FWIW, Simpson's paradox is a "veridical" paradox, not a "vertical" paradox as noted in the article. Apparently veridical isn't yet accepted by spelling checkers. (As I'm writing, Chrome offers the single suggestion "vertical" for my "veridical".)
xerophtye 13y ago
Those interactive charts and graphs!! (My first encounter with vudlab and am loving it!)

210 points by mmaia 13y ago

sp332 13y ago

Reminds me of Anscombe's Quartet: four datasets that have the same mean, variance, correlation, and linear regression, but are really very different. http://en.wikipedia.org/wiki/Anscombe%27s_quartet http://en.wikipedia.org/wiki/Anscombe%27s_quartet

lewis500 13y ago
maybe we'll do that one next!

tel 13y ago

Simpson's Paradox is scary if you've not seen it before, but you should not stop here and instead proceed immediately on to omitted variable bias and the conversation here (http://normaldeviate.wordpress.com/2013/06/20/simpsons-paradox-explained/ http://normaldeviate.wordpress.com/2013/06/20/simpsons-parad...). In short, Simpson's paradox occurs because probability distributions and causal claims are distinct things which behave differently. It's nothing more than a particular, insidious example of correlation not implying causation. It's perfectly possible for a probability distribution to have a "contradictory" shape, but perfectly impossible for logical statements about the world to be contradictory. The resolution is that you shouldn't let your probability distributions turn into logical statements without analyzing your causal assumptions. This will lead you to whether or not excluding a variable is omitted variable bias (and whether including an improper one will lead to included variable bias, which is rarely recognized).

tlb 13y ago

The Omitted Variable Problem is part of my mental framework that causes me to not believe most epidemiological studies, especially ones that confirm a popular belief. The almost universally omitted variable is health-consciousness. Some people are health-conscious and some aren't. People who are health-conscious do a whole bunch of things, some of which help (like exercise, sleep well, eat moderately). They also do things that are widely believed to be good for you, like eating broccoli. So if you do a study, you'll find that people who eat lots of broccoli are healthier. You'll be able to confirm pretty much any widely believed health folk wisdom, unless it's something quite harmful, as long as you omit health-consciousness as a variable.

maxerickson 13y ago
If it was a huge deal, wouldn't epidemiologists be slobbering over the statistical power they would get from using it? That is, if they can control for it, they can make assertions about smaller effects.
- gwern 13y ago
  > If it was a huge deal, wouldn't epidemiologists be slobbering over the statistical power they would get from using it? Why would you want to do that? You can build a nice career on publishing (spurious) associations. As long as relevant variables are omitted, no epidemiologist need ever be unemployed.
  - maxerickson 13y ago
    I'm optimistic enough to believe in a few wild cards trying to do good science. A whole bunch of the rest have to put a little bit of effort into competing with them. It occurred to me a while after my other comment that a belief in the ability of people that test as health conscious to actually do things that improve their health might not be especially justified. I mean, lots of people are quite happy with whatever quackery, maybe it really does all come out in the wash.
  - maxander 13y ago
    That may get you written about in popular-science magazines, but your work will have a low impact factor- this is largely measured in how many of your fellow researchers cite your work, and its crucial to an academic researcher's career. If your colleagues can easily see that your statistical analysis is flawed, they won't waste their own time doing work that builds off of yours.
    - wnewman 13y ago
      It is not necessarily true that just because they are called scientists, real people with real motives, funded by organizations of real people with real motives, operating in a political regime controlled by real people with real motives, will act that way. (By "that way" I mean "your work will have a low impact factor- this is largely measured in how many of your fellow researchers cite your work" and "If your colleagues can easily see that your statistical analysis is flawed, they won't waste their own time doing work that builds off of yours.") See e.g. Chan and Boliver "The Grandparents Effect in Social Mobility", http://asr.sagepub.com/content/78/4/662 http://asr.sagepub.com/content/78/4/662 (HT http://www.arnoldkling.com/blog/a-grandparent-effect/ http://www.arnoldkling.com/blog/a-grandparent-effect/) and many of the works that it cites. There seems to be a sizable mutually-citing "scientific" literature devoted to studying correlation in biological descent while dogmatically not controlling for ordinary DNA-based inheritance of psychological traits from parent to child. (Indeed, not even controlling for DNA-based inheritance of physical traits. Health and height and attractiveness are correlated with income, and are significantly physically heritable.) By not controlling for the properties which are transmitted by sperm meeting egg, your can find all sorts of fascinating possibilities to investigate to explain the correlation. (E.g. Chan and Boliver carefully note that "well-connected grandparents could also use their social contacts to help grandchildren with job searches." Good, good, carefully investigate all the possibilities.)

teamonkey 13y ago

The point of an interactive illustration like this is that it should make it more intuitive and easier to understand than the core concept, not more confusing. What is the meaning of the green and purple lines on the graph, why do they have different gradients and why can't I adjust them? Why does the Simpson's Paradox apply sometimes and not others? Why are there so many bars and donut charts? What does the gray circle around the donuts mean? Information overload.

publicfig 13y ago
I feel as thought both graphs with purple and green lines are accurately explained. Were you trying to skim through the article or were you still left confused after reading it? The top one is explained in the accompanying text and the bottom one is explained with a labeled x and y axis and relies on the information provided above.
- teamonkey 13y ago
  If the article describes the subject adequately, what is the point of the graphic? I found the interactive graphic more confusing than the description. Compare and contrast to the clarity of their central limit theorem demonstation [1] vs its wikipedia page [2]. [1] http://blog.vctr.me/posts/central-limit-theorem.html http://blog.vctr.me/posts/central-limit-theorem.html [2] http://en.wikipedia.org/wiki/Central_limit_theorem http://en.wikipedia.org/wiki/Central_limit_theorem
  - publicfig 13y ago
    By that, I meant that the article described what the graph was representing, not that the article described the information presented adequately. Also, I absolutely disagree with the assumption you made that graphs shouldn't be present if the information is presented in the article. It's good to provide visual demonstrations when applicable in order to help clarify subjects.
jere 13y ago
It took a little playing around to figure out some of the graphics. The donuts indicate which group (men or women) was admitted more in each kind of department. Play around with the sliders and you can see it toggle in the combined row, while simultaneously a yes/no answer updates after the question "Simpson's paradox?" It would really help if yes/no was highlighted. Basically, you can make the paradox appear by dropping the % of women applying to easy departments. You can't make the donut switch in easy or hard departments individually... that's the whole point: women have a slight advantage in each. Honestly, I found the "Proper Pooling" section to have the most intuitive explanation. Combined, there is a bias. By department, there is no bias. The reason is clear: very few women applying to easier departments.

maxk42 13y ago

That is some sexy javascript.

ZoF 13y ago
My first thought as well
vicapow 13y ago
thank you! :D

trains 13y ago

It's nice visually but the text-based information could be more thorough - which would result in less links at the bottom.The more info contained on the page the better!

alexPetrov 13y ago

Reading up on Simpson's Paradox again made me realize something: women retrieving custody more often than men appears to me to be a perfect example of Simpson's Paradox. Overall, women get custody more often than men, this is true, but if you consider only the cases where men actually asked for or attempted to retrieve custody, this is no longer true[0]. [0]: http://www.villainouscompany.com/vcblog/archives/2012/04/child_supportcu.html http://www.villainouscompany.com/vcblog/archives/2012/04/chi...

Tichy 13y ago
How to interpret that, though? Seems likely to me that who asks for custody would be influenced by their chances of receiving it. How many of those cases are "contested custody"? Then that article places the claim "Additional evidence, however, indicates that women may be less able to afford the lawyers and experts needed in contested custody cases (see “Family Law Overview”) and that, in contested cases, different and stricter standards are applied to mothers." without providing any data. So it ends up being a propaganda piece, which makes it not very trustworthy.

dsego 13y ago

For me this was far better explained in the following text (also posted on HN not so long ago): http://www.michaelnielsen.org/ddi/if-correlation-doesnt-imply-causation-then-what-does/ http://www.michaelnielsen.org/ddi/if-correlation-doesnt-impl...

tensafefrogs 13y ago

Anyone interested in this stuff should definitely watch the lectures from the Stats 110 class from Harvard that's up on iTunes U (and perhaps other places). Lecture 6 talks about this paradox and if I recall correctly he might even talk about this exact case: https://itunes.apple.com/us/course/statistics-110-probability/id502492375 https://itunes.apple.com/us/course/statistics-110-probabilit...

jasonwatkinspdx 13y ago

Judea Pearl has formalized a resolution to Simpson's Paradox: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.34.8955&rep=rep1&type=pdf http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.34.... An oversimplification of his idea would be to say that given assumptions about the causal independence of variables, it becomes clear which way you should group the data. Although it's not always possible to make these independence assumptions, much of the time they are obvious and uncontroversial. Taking the Berkeley gender bias case as an example: We know it's possible that biological gender influences which department graduates apply to, but that it's impossible that the department a graduate applies to influences their biological gender. This fact alone tells us that we need to look at the data by department rather than in aggregate, resolving the paradox.

gweinberg 13y ago
Well, it certainly makes more sense to look at the by department data rather than the aggregate data. But I would say that the fact that we see such large disparities in which departments are applied to is already enough to refute the idea that male and female applicants are pretty much the same, and that therefore it would be pretty reckless to conclude any sex discrimination based on the difference in acceptance rates.
- deleted 13y ago
  [deleted]
- tel 13y ago
  And Pearl's point is that the "real world" logic you just applied is both important and not statistical—you need to augment your analysis with these causal assumptions in order to translate probability into meaningful causal statements.
tel 13y ago
That's a great reference. I'm going to use it from now on too.
- 13y ago

cfontes 13y ago

Really interesting text, thanks. But I think this last statement "or Texas schools to waste money copying Wisconsin." was meant to be the other way around

yen223 13y ago
I may have read it wrong, but I think the statement is correct. Texas minorities outperformed Wisconsin's, but the stats made it look like Wisconsin's minorities did better.
- penrod 13y ago
  To elaborate: Black students in Texas outperform black students in Wisconsin, hispanic students in Texas outperform hispanic students in Wisconsin, and white students in Texas outperform white students in Wisconsin. Overall test scores are higher in Wisconsin only because white students in both states are the highest scoring group, and Wisconsin schools have a higher proportion of white students than Texas schools. To the extend that any of this can be copied, it would probably be better for Wisconsin to improve the test scores of its ethnic groups to Texan levels, rather than for Texas to emulate Wisconsin's lilly-whiteness.

martimoose 13y ago

I had a hard time understanding the last graphic, because I was reading on an iPad, and the sliders don't work. I thought it was all static, and couldn't match the numbers in the graph with the previously mentioned numbers, as the defaults of the sliders do not give a simpson's paradox.

wunderlust 13y ago

FWIW, Simpson's paradox is a "veridical" paradox, not a "vertical" paradox as noted in the article. Apparently veridical isn't yet accepted by spelling checkers. (As I'm writing, Chrome offers the single suggestion "vertical" for my "veridical".)

xerophtye 13y ago

Those interactive charts and graphs!! (My first encounter with vudlab and am loving it!)