2 ms·

Simpson's Paradox keeps experimenters up at night because it embodies the idea that although your data might say one thing, it's always possible that slicing th

Simpson's Paradox keeps experimenters up at night because it embodies the idea that although your data might say one thing, it's always possible that slicing the data via some unknown axis of finer granularity might paint a very different picture. It's hard to know if there is such an axis lurking there in your data, let alone, what it might be. If you get paranoid about its presence it can lead you to second guess pretty much every statistic. "I know that 4 out of 5 dentists recommend chewing X Brand gum but what if I slice the dentists by number of eyes? Maybe both one-eyed dentists and two-eyed dentists aren't so enthusiastic."

grobbyy 2y ago
It does, but only experiments who don't have a strong grounding in research methods. Simpson's paradox is part of a broader problem: correlation does not imply causation. In practice, it's one of many problems with making decisions based on correlations. In a randomized control trial, with a large random sample, the odds of Simpson's Paradox coming up are astronomically low. Good statisticians WILL second-guess ANY conclusions based on post-hoc data analysis. To frame this in scientific jargon, exploratory data analysis and correlations are great at generating hypotheses, but those need to be confirmed with methods appropriate for confirmatory analysis.