5 ms·
Everything is correlated (2014–23)
- 2rsf 1y agoDid they quote https://www.tylervigen.com/spurious-correlations https://www.tylervigen.com/spurious-correlations ?
- eisvogel 1y agoIt's just as I suspected - there are NO coincidences.
- andsoitis 1y agothere is but a single unfolding, and everything is part of it
- dang 1y agoCorrelated. Others? Everything Is Correlated - https://news.ycombinator.com/item?id=19797844 https://news.ycombinator.com/item?id=19797844 - May 2019 (53 comments)
- stouset 1y agoCorrelated, you mean?
- pnt12 1y agoThose would be all articles posted in HN :)
- dang 1y agoThat's so good I had to nick it for the parent comment. Thanks! (It said "Related" before, of course: https://hn.algolia.com/?dateRange=all&page=0&prefix=true&query=by%3Adang%20related%20others&sort=byDate&type=comment https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que....)
- senko 1y agoThe article missed the chance to include the quote from that standard compendium of information and wisdom, The Hitchhiker's Guide to the Galaxy: > Since every piece of matter in the Universe is in some way affected by every other piece of matter in the Universe, it is in theory possible to extrapolate the whole of creation — every sun, every planet, their orbits, their composition and their economic and social history from, say, one small piece of fairy cake.
- sayamqazi 1y agoWouldnt you need the T_zero configuration of the universe for this to work? Given different T_zero configs of matter and energies T_current would be different. and there are many pathways that could lead to same physical configuration (position + energies etc) with different (Universe minus cake) configurations. Also we are assuming there is no non-deterministic processed happening at all.
- senko 1y agoI am assuming integrating over all possible configurations would be a component of The Total Perspective Vortex. After all, Feynman showed this is in principle possible, even with local nondeterminism. (this being a text medium with a high probability of another commenter misunderstanding my intent, I must end this with a note that I am, of course, BSing :)
- eru 1y ago> Wouldnt you need the T_zero configuration of the universe for this to work? Why? We learn about the past by looking at the present all the time. We also learn about the future by looking at the present. > Also we are assuming there is no non-deterministic processed happening at all. Depends on the kind of non-determinism. If there's randomness, you 'just' deal with probability distributions instead. Since you have measurement error anyway, you need to do that anyway. There are other forms of non-determinism, of course.
- psychoslave 1y ago
- petters 1y agoIf two things e.g. both change over time, they will be correlated. I think it can be good to keep this article in mind
- eru 1y ago> If two things e.g. both change over time, they will be correlated. No? You can have two independent random walks. Eg flip a coin, gain a dollar or lose a dollar. Do that to times in parallel. Your two account balances will change over time, but they won't be correlated.
- gwern 1y agohttps://gwern.net/doc/statistics/causality/1926-yule.pdf https://gwern.net/doc/statistics/causality/1926-yule.pdf
- vi0g0d 1y ago[dead]
- Evidlo 1y agoThis is such a massive article. I wish I had the ability to grind out treatises like that. Looking at other content on the guy's website, he must be like a machine.
- tmulc18 1y agogwern is goated
- kqr 1y agoIIRC Gwern lives extremely frugally somewhere remote and is thus able to spend a lot of time on private research.
- tux3 1y agoIIRC people funded moving gwern to the bay not too long ago.
- lazyasciiart 1y agoThat and early bitcoin adoption. There’s a short bio somewhere on the site.
- pas 1y agolots of time, many iterations, affinity for the hard questions, some expertise in research (and Haskell). oh, and also it helps if someone is funding your little endeavor :)
- aswegs8 1y agoI wish I would be even able to read things like that.
- apples_oranges 1y agoPeople didn't always use statistics to discover truths about the world. This, once developed, just happened to be a useful method. But given the abuse using those methods, and the proliferation of stupidity disguised as intelligence, it's always fitting to question it, and this time with this correlation noise observation. Logic, fundamental knowledge about domains, you need that first. Just counting things without understanding them in at least one or two other ways, is a tempting invitation for misleading conclusions.
- mnky9800n 1y agoThere is a quote from George Lucas where he talks about how when new things come into a society people have a tend to over do it. https://www.youtube.com/watch?v=VEIrQUXm_hY https://www.youtube.com/watch?v=VEIrQUXm_hY
- apples_oranges 1y agoNice, yeah. With many movies one has to ask: What's the point? Especially all Disney Star Wars..
- arduanika 1y agoQuoted repeatedly by Mr. Plinkett in his new drop yesterday. https://www.youtube.com/watch?v=0xeMak4RqJA https://www.youtube.com/watch?v=0xeMak4RqJA
- kqr 1y ago> People didn't always use statistics to discover truths about the world. And they were much, much worse off for it. Logic does not let you learn anything new. All logic allows you to do is restate what you already know. Fundamental knowledge comes from experience or experiments, which need to be interpreted through a statistical lens because observations are never perfect. Before statistics, our alternatives for understanding the world was (a) rich people sitting down and thinking deeply about how things could be, (b) charismatic people standing up and giving sermons on how they would like things to be, or (c) clever people guessing things right every now and then. With statistics, we have to a large degree mechanised the process of learning how the world works, and anyone sensible can participate, and they can know with reasonable certainty whether they are right or wrong. It was impossible to prove a philosopher or a clergyman wrong! That said, I think I agree with your overall point. One of the strengths of statistical reasoning is what's sometimes called intercomparison, the fact that we can draw conclusions from differences between processes without understanding anything about those processes. This is also a weakness because it makes it easy to accidentally or intentionally manipulate results.
- st-keller 1y ago„This renders the meaning of significance-testing unclear; it is calculating precisely the odds of the data under scenarios known a priori to be false.“ I cannot see the problem in that. To get to meaningful results we often calculate with simplyfied models - which are known to be false in a strict sense. We use Newtons laws - we analyze electric networks based on simplifications - a bank-year used to be 360 days! Works well. What did i miss?
- whyever 1y agoIt's a quantitative problem. How big is the error introduced by the simplification?
- bjornsing 1y agoThe problem is basically that you can always buy a significant result with money (large enough N always leads to ”significant” result). That’s a serious issue if you see research as pursuit of truth.
- syntacticsalt 1y agoReporting effect size mitigates this problem. If observed effect size is too small, its statistical significance isn't viewed as meaningful.
- bjornsing 1y agoSure (and of course). But did you see the effect size histogram in the OP?
- syntacticsalt 1y agoAre you referring to the first figure, from Smith, et al, 2007? If so, I couldn't evaluate whether gwern's claim makes sense without reading that paper to get an idea of, e.g., sample size and how they control for false positives. I don't think it's self-evident from that figure alone. One rule of thumb for interpreting (presumably Pearson) correlation coefficients is given in [0] and states that correlations with magnitude 0.3 or less are negligible, in which case most of the bins in that histogram correspond to cases that aren't considered meaningful. [0]: https://pmc.ncbi.nlm.nih.gov/articles/PMC3576830/table/T1/ https://pmc.ncbi.nlm.nih.gov/articles/PMC3576830/table/T1/
- syntacticsalt 1y agoI don't disagree with the title, but I'm left wondering what they want us to do about it beyond hinting at causal inference. I'd also be curious what the author thinks of minimum effect sizes (re: Implication 1) and noninferiority testing (re: Implication 2).
- jongjong 1y agoAlso, I'm convinced that the reason humans intuitively struggle to figure out causality is because the vast majority of causes and effects are self-reinforcing cycles and go both ways. There was little evolutionary pressure for us to understand the concept of causality because it doesn't play a strong role in natural selection. For example, eat a lot and you will gain weight, gain weight and you will feel more hungry and will likely eat more. Or exercise more and it becomes easier to exercise. Earning money becomes easier as you have more money. Public speaking becomes easier as you do it more and the more you do it, the easier it becomes. Etc...
- ctenb 1y ago> Public speaking becomes easier as you do it more and the more you do it, the easier it becomes. That's saying the same thing twice :)
- jongjong 1y agoHaha yes. I meant to say the more public speaking you do, the easier it gets so the more often you want to do it.
- renox 1y ago> Or exercise more and it becomes easier to exercise. Only if you don't injure yourself while exercising.
- jongjong 1y agoA lot of things can happen to break a self-perpetuating cycle. But it's usually some extreme event. The cycle keeps optimizing in a particular direction and this eventually leads to an extreme situation which becomes unsustainable. The natural equilibrium of a self-reinforcing cycle is not static but drifting towards some extreme unstable state. There is usually a point where it breaks down. But I suspect that being able to figure out causation doesn't matter much from a survival or reproduction perspective because cause and effect are just labels. Reality in a self-perpetuating cycle is probably like Condition A is 70% responsible and Condition B is 30% responsible for a problem but they feedback and exacerbate each other... You could argue that Condition A is the cause and Condition B is the effect because B < A but that's not quite right IMO. Also, it's not quite right to say that because A happened first, that A is the cause of a severe problem... The problem would never have gotten so bad to such extent without feedback from B.
- 01HNNWZ0MV43FF 1y ago> For example, while looking at biometric samples with up to thousands of observations, Karl Pearson declared that a result departing by more than 3 standard deviations is “definitely significant.” Wait. Sir Arthur Conan Doyle lived at basically the exact same time as this Karl Pearson. Is that why the Sherlock Holmes stories had handwriting analysis so frequently? Was there just pop science going around at the time that like, let's find correlations between anything and anything, and we can see that a criminal mastermind like Moriarty would certainly cross their T's this way and not that way?
- cluckindan 1y agoI wonder if this tendency to correlate truly holds for everything? Intuitively it more or less demonstrates that nature tends to favor zero-sum games. Maybe analyzing correlations within the domain of theoretical physics would highlight true non-correlations in some particular approaches? (pun only slightly intended)
- eru 1y ago> Intuitively it more or less demonstrates that nature tends to favor zero-sum games. Please explain.
- cluckindan 1y agoFor every action, there is an opposite and equal reaction. For example, there is a correlation between the acceleration and deceleration of colliding objects: inertia is transferred, not created or destroyed. Similarly, for every chemical and nuclear reaction, when something is gained, something else is lost. For example, when two ions bond covalently by sharing electrons, a new molecule is gained, but the two ions are no longer what they previously were. So there is a correlation between gain of reaction products and loss of reactants. But perhaps such analogies cannot be found everywhere in theoretical physics. Perhaps such a non-correlation would be a sign of a novel discovery, or a sign that a theory is physically invalid. It could be a signal of something for sure.
- terminalbraid 1y agoHow do I reconcile this with "entropy invariably increases" which is a contradiction to your hypothesis that "nature tends to favor zero-sum games"? How do I reconcile "for every chemical and nuclear reaction, when something is gained, something else is lost" with catalysts increasing rate but not being consumed themselves? In fact you can show there are an uncountably infinite number of broken symmetries in nature, so it is mathematically possible to concoct a parallel number of cases where nature does not have some "zero sum game" by Noether's Theorem. Your statement is just cherry picking a few and then (uncountably infinitely) overgeneralizing.
- simsla 1y agoThis relates to one of my biggest pet peeves. People interpret "statistically significant" to mean "notable"/"meaningful". I detected a difference, and statistics say that it matters. That's the wrong way to think about things. Significance testing only tells you the probability that the measured difference is a "good measurement". With a certain degree of confidence, you can say "the difference exists as measured". Whether the measured difference is significant in the sense of "meaningful" is a value judgement that we / stakeholders should impose on top of that, usually based on the magnitude of the measured difference, not the statistical significance. It sounds obvious, but this is one of the most common fallacies I observe in industry and a lot of science. For example: "This intervention causes an uplift in [metric] with p<0.001. High statistical significance! The uplift: 0.000001%." Meaningful? Probably not.
- V__ 1y agoI really like this video [1] from 3blue1brown, where he proposes to think about significance as a way to update the probability. One positive test (or in this analog a study) updates the probability by X % and thus you nearly always need more tests (or studies) for a 'meaningful' judgment. [1] https://www.youtube.com/watch?v=lG4VkPoG3ko https://www.youtube.com/watch?v=lG4VkPoG3ko
- jpcompartir 1y ago^ And if we increase N enough we will be able to find these 'good measurements' and 'statistically significant differences' everywhere. Worse still if we did not agree in advance what hypotheses we were testing, and go looking back through historical data to find 'statistically significant' correlations.
- ants_everywhere 1y agoWhich means that statistical significance is really a measure of whether N is big enough
- kqr 1y agoThis has been known ever since the beginning of frequentist hypothesis testing. Fisher warned us not to place too much emphasis on the p-value he asked us to calculate, specifically because it is mainly a measure of sample size, not clinical significance.
- hshshshshsh 1y agoDoesn't everything means all things that exist in universe and since they exist in same universe they are correlated?
- terminalbraid 1y agoCausality from relativity prevents this from being generally true. There could be things in this universe sufficiently far away that we cannot see because whatever interaction we could observe from it has not reached here yet.
- frotaur 1y agoIt is actually inflation that precludes this from being true (along with relativity, of course). If there wasn't a period where spacetime inflated extremely rapidly, given two points, no matter how far apart, their causal pasts would eventually intersect, if going sufficiently back in the past. As such, they could correlate through a common cause C, which lies in both of their causal pasts.
- nathan_compton 1y agoReally classic "rationalist" style writing: a soup of correct observations about statistical phenomena with chunks of weird political bullshit thrown in here and there. For example: "On a more contemporary note, these theoretical & empirical considerations also throw doubt on concerns about ‘algorithmic bias’ or inferences drawing on ‘protected classes’: not drawing on them may not be desirable, possible, or even meaningful." This is such a bizarre sentence. The way its tossed in, not explained in any way, not supported by references, etc. Like I guess the implication being made is something like "because there is a hidden latent variable that determines criminality and we can never escape from correlations with it, its ok to use "is_black" in our black box model which decides if someone is going to get parole? Ridiculous. Does this really "throw doubt" on whether we should care about this? The concerns about how models work are deeper than the statistical challenges of creating or interpreting them. For one thing, all the degrees of freedom we include in our model selection process allow us to construct models which do anything that we want. If we see a parole model which includes "likes_hiphop" as an explanatory variable we ought to ask ourselves who decided that should be there and whether there was an agenda at play beyond "producing the best model possible." These concerns about everything being correlated actually warrant much more careful understanding about the political ramifications of how and what we choose to model and based on which variables, because they tell us that in almost any non-trivial case a model is at least partly necessarily a political object almost certainly consciously or subconsciously decorated with some conception of how the world is or ought to be explained.
- zahlman 1y ago> This is such a bizarre sentence. The way its tossed in, not explained in any way, It reads naturally in context and is explained by the foregoing text. For example, the phrase "these theoretical & empirical considerations" refers to theoretical and empirical considerations described above. The basic idea is that, because everything correlates with everything else, you can't just look at correlations and infer that they're more than incidental. The political implications are not at all "weird", and follow naturally. The author observes that social scientists build complex models and observe huge amounts of variables, which allows them to find correlations that support their hypothesis; but these correlations, exactly because they can be found everywhere, are not anywhere near as solid evidence as they are presented as being. > Like I guess the implication being made is something like "because there is a hidden latent variable that determines criminality and we can never escape from correlations with it, its ok to use "is_black" in our black box model which decides if someone is going to get parole? No, not at all. The implication is that we cannot conclude that the black box model actually has an "is_black" variable, even if it is observed to have disparate impact on black people.
- nnnnico 1y agoThere is no zero in the real world
- pcrh 1y agoThis is why experimental science is different from observational studies. Statistical analyses provide a reason to believe one hypothesis over another, but any scientist will extend that with an experimental approach. Most of the examples given in this blog post refer to medical, sociological or behavioral studies, where properly controlled experiments are hard to perform, and as such are frequently under-powered to reveal true cause-effect associations.
- deleted 1y ago[deleted]
- pacbard 1y agoThis was my take as well. At least microeconomics has moved away from large-scale observational studies and has moved into experimental and quasi-experimental studies. While the methods alone cannot fix it all ("You can’t fix by analysis what you bungled by design" [1] after all), it gets somewhat closer to unbiased results. [1]: https://www.degruyterbrill.com/document/doi/10.4159/9780674040267-001/html?lang=en https://www.degruyterbrill.com/document/doi/10.4159/97806740...
- psychoslave 1y agoLooks like an impressive thorough piece of investigation. Well done. That said, holistic supposition can certainly be traced back as far as writting dawns. Here the focus on more modern/contemporary era is legitimate to keep the focus delimited on a more specific concern, but is a bit obfuscating this fact. Maybe it's already acknowledged in the document, I read it all yet.
- justonceokay 1y agoStatistical correlations anre important to establish but there are the easiest and least useful part of the research. Creating theories as to “why” and “how” these correlations exist are what advances our knowledge. I read lot of papers that painstakingly show a correlation in the data, but then their theory about the correlation is a complete non sequitur.
- endymion-light 1y agothe rest of the page has amazing design, but there's just something about the graphs switching from dark to light that flashbangs my eyes really badly - i think it's the sudden light!
- gwern 1y agoThat is unintentional and a bug in the dark-mode. For dark-mode, we rely on https://invertornot.com/ https://invertornot.com/ to dynamically decide whether to fade or negate/invert. (Background: https://gwern.net/invertornot https://gwern.net/invertornot ) The service uses a small NN and is not always correct, as in these cases. Sorry. I have filed them as errors with InvertOrNot, and will manually set them to invert.
- endymion-light 1y agoAh lovely! Not a critique as enjoyed the rest of the article, just wanted to comment in case it helped resolve it!
- derbOac 1y agoArguments like this have been around for decades. I think it's important to keep in mind — critical even. At the same time, as I've been forced to wrestle with it more in my work, I've increasingly felt that it's sort of empty and unhelpful. "Crud" does happen in patterns, like a kind of statistical cosmic background radiation — it's not meaningless. Sometimes it's important to understand it, and treating it as such gets no one anywhere. Sometimes the associations are difficult to explain easily when you try to pick it apart, and other times I think they're key to understanding uncontrolled confounds that should be controlled for. As much as this background association is present too, it's not always there. Sometimes things do have zero association. Also, trying to come up with a "meaningful" effect size that's not zero is pretty arbitrary and subjective. There's probably more productive ways of framing the phenomenon.
- ricardobayes 1y agoNot commenting on the topic at hand, but my goodness, what a beautiful blog. That drop cap, the inline comments on the right hand side that appear on larger screens, the progress bar, chef's kiss. This is how a love project looks like.
- scarmig 1y agoYou may be interested in gwern's dropcap article: https://gwern.net/dropcap https://gwern.net/dropcap
- alexpotato 1y agoI was recently looking at a large timeseries dataset. I noticed when doing a scatter plot of two variables and noticed that there were several "lines" of dots. This generally implies that subsets of the two variables may have correlations or there is a third variable to be added. I did some additional research and it is possible for two variables with large N to show correlation for short bursts even if both variables are random. I mention for two reasons: 1. I was just doing the above and saw the OP article today 2. Despite taking multiple college level stats classes, I don't remember this ever being mentioned.
- bsoles 1y agoAny two linearly increasing or decreasing datasets will be necessarily highly correlated. But that doesn't mean anything unless you also have a plausible explanation why it is so. Also correlation is often taken to mean linear correlation. For example, you can have two nonlinearly varying dataset to be perfectly (rank) correlated while their linear correlation can be close to zero. People often attach undue meaning to correlation.
- scoofy 1y agoA really interesting essay. I'm genuinely surprised the author didn't delve more into philosophy here. I think the biggest issue this essay illustrates is the inductive-deductive jump that modeling (especially mathematical modeling makes). That is, if the axioms we are using are correct, then the output of the model should reflect reality. This is very obviously not always the case, and because of the black swan problem (in a very real sense) we can never really know if it does. Any time we wade into solipsism, however, I think it's important to remember that statistical analysis is a tool, not an arbiter of Truth. We are trying to exist in the mess of a world we live in, and we're going to be using every possible advanced tool we have in our arsenal to do that, and the standard model of science is probably the best tool we have. At the same time, we should often return to that solipsism an remember that where we can improve the model, we should.
- doommood 1y agoDoes the author account for time autocorrelation? Pearson’s correlation assumes independence between observations, yet time series data often violate this assumption. Without adjusting for autocorrelation, correlations may be artificially inflated.
- jhhh 1y agoI mouse over the first graphic in dark mode and get flashbanged with a white diagram. Hover text obscures the axis text. I click the diagram hoping to read the text. In the expanded picture the alt text now obscures the axis instead. Close the picture and read on to the starting section. A dropped capital. It looks like an 'R' with some flourish. 'Rnowing'. Ok, it must be a 'k' based on context, but appears to resemble a lower case cursive 'k' and not a capital 'K'. Dropping the capital letter to lowercase in a drop cap. A minuscule pun one character in. Off to a good start. I continue reading to the second diagram. Fearful from my previous encounter I rest my cursor on what looks like a safe zone: 'click to expand'. The section immediately unfurls and pushes the diagram off the page. 'Everything is correlated' I tell myself. If there's a hover event there must be an 'on-not-hover' event too, right? I move my cursor to an empty zone. Nothing happens. I click in the empty space to remind the browser my cursor is not hovering. Nothing. 'Only some things are correlated' I think, closing the section manually. Eventually I see a hovernote 4 and think to myself 'it's nice these notes won't interrupt the flow of reading' as I click on it. The page shifts to show text already on the screen and I lose my place briefly. The word 'error' is now partially obscured by the footnote bounding box. I chuckle to myself. Maybe there is an option for reader mode I think as I start scrolling up. A gear reveals itself in the top corner. As I scroll down the options I see a 'Auto Reader Mode'. Exactly what I need. I move my mouse over to the text. This was a poor choice. The menu retracts and now that my mouse is partially over a hover note it scrolls itself fully into view. 'Tibshirani 2014' is revealed.
- roncesvalles 1y agoAnd that, kids, is why A/B testing is nonsense.