9 ms·
How to Think about Correlation?
- adamnemecek 6y agoCorrelation and convolution are adjoints.
- bollu 6y agoCan you expand a little further? I don't see the adjunction (between what categories?)
- codeplay 6y agoi GUESS what he means is something like inverse of covariance matrix gives correlation matrix, something like this question: https://stats.stackexchange.com/q/140080 https://stats.stackexchange.com/q/140080
- contravariant 6y agoIt's somewhat weird to see someone immediately jump to category theory when it's more likely to be about basic linear algebra. That said convolution and correlation are not linear operators but bilinear operators, so I'm struggling a bit to see in what sense they're adjoint. Maybe something like: <Conv(x,y), z> = <y, Corr(x, z)> which does actually work if you let the inner product be the integral of the product and the correlation the time-correlation (so the inner product is the correlation at time 0).
- adamnemecek 6y agoThat is what I was referring to yes, however it is also adjoints in the category theory sense. http://www.reproducibility.org/RSF/book/bei/conj/paper_html/index.html http://www.reproducibility.org/RSF/book/bei/conj/paper_html/...
- bollu 6y agoAh I see; The word "adjoint" tripped me up, I presumed they meant "adjunction", and not the matrix-transpose-adjoint :)
- jph 6y agoThe article's question: "Why are correlations meaningful?" Here's a simplified answer. Correlations are meaningful because they indicate how strongly variables can predict other variables, in some circumstances. This is valuable because some variables are easier to use than others. A positive correlation between two variables means they tend to move in the same direction (relative to their mean). A negative correlation between two variables means they tend to move in opposite directions. A strong correlation between two variables means that they tend to change in much the same kinds of relative magnitudes, and also that either variables can be used to generally predict the other. A weak correlation between two variables means that the two variables tend to change in somewhat the same kinds of relative magnitudes, sometimes with some kinds of exceptions, and also that either variable can be used to somewhat predict the other. Thus a strong positive correlation means that variables can predict each other effectively, and also move in the same direction. This is especially valuable for projects where one variable is easy to obtain, yet the other variable is hard to obtain. And it turns out that a strong positive correlation can be a good place to hunt for possible causality, such as hunting for something that causing both variables to change together. However, correlation doesn't imply causation. A strong positive correlation can be good evidence that a hypothesized causality is likely to be true. For example an experiment can test if X causes Y, by varying X, and discovering that X and Y tend to change together. Edit: the replies below are great-- thank you-- well worth reading. I'm editing my post to make it clearer that it's a simplified answer, and to add suggestions by the replies.
- Galanwe 6y agoI don't quite like this answer, because it simplifies a lot of things about correlations which are just not really accurate. > Corrleations are meaningful because they indicate how strongly variables can predict other variables. This is not wrong per se, but may be wrongly interpreted. 1) I think it is important to stress that the reverse is not true. If a correlation is 0, it does not mean the variables are independent. It means they are linearly independent. There are a lot of things out there which are very much linked but not with a linear relationship. 2) This is only true if the correlation was done properly. For instance, correlating non-stationary variables will spuriously yield a high correlation, giving a false impression of link. The classic example is trying to correlate stock prices between themselves. An other example of spurious correlation is when not enough points are given to get an accurate result. Looking at tstat, r2, etc. is as important as looking at the actual correlation coefficient. 3) You use the word "predict" here, which I don't quite like. Correlation can predict outcomes only in the case where you have proven that past information will continue to occur. This is not always the case. > A positive correlation between two variables means they tend to move in the same direction. This is misleading also. The variables tend to move in the same direction _relatively to their mean_! I can plot you very different looking curves that are highly correlated just because their mean vary differently. Especially because of potential outliers. > And it turns out that a strong positive correlation can be a good place to hunt for possible causality, such as hunting for something that causing both variables to change together. I tend to dislike that thought process. It is very tempting to infer causality from correlation. Especially when automating correlation at scale is affordable, so generating spurious correlation is easy. To me correlation is more a way to reinforce a hypothesis of causality that is already suspected. It won't give you the direction, but may hint you that you are on the right track. I mean, correlation should not be a clue to discover causality, as there are too many false positives. Correlation should be an intermediary step in studying a potential causality.
- dr_dshiv 6y agoImagine my surprise when I realized that correlation and synchrony are (almost) the same thing.
- adamnemecek 6y agoWhat do you mean by synchrony?
- dr_dshiv 6y agoPhase synchrony, where two signals rise and fall together. Cross correlation measures out of phase synchrony
- j7ake 6y agoJust to add another reason to think of correlation as linear models: spearman correlation is analogous to fitting a linear model on the ranks of the inputs rather than using the values themselves.
- max_ 6y ago"Statisticians will tell you that mud causes rain" — Yudea Pearl The main problem of correlation is that researchers often "start with a conclusion and then fill in the blanks". The thing is that several items can fill in these blanks. With coherence. Making predictions based on these techniques is often misleading & in some cases even dangerous. You can attribute the irreplicability of several "scientific findings" to the use of correlation. Correlation is great for " analysis" but not a way for drawing a conclusion
- hbarka 6y agoIf R-squared is 10% (and p-value is 0.01), does it mean the correlation is 10% better than random?
- DavidSJ 6y agoR^2 of 10% means 10% of the variance in one variable is accounted for by the linear regression on the other variable. That is, the expectation of the conditional variance (the amount of variance left after using the information from the linear regression) is 90% of the unconditional variance. A p-value of 0.01 means that under the null hypothesis (generally that the variables are independent), a result at least this extreme (in this case, with at least this large a correlation) would be seen 1% of the time. Note in general that R^2 = 10% does not imply p = 0.01, nor the converse. They are largely unrelated measures. In principle one could have a small correlation with high significance, or a large correlation with low significance.
- sieste 6y agoThere are at least 13 ways to think about correlation Rodgers & Nicewander 1988 "Thirteen ways to look at the correlation coefficient" https://www.jstor.org/stable/pdf/2685263.pdf https://www.jstor.org/stable/pdf/2685263.pdf
- Geminidog 6y agoCorrelations are a profound part of our universe. When observing the universe, humans can never prove any facts about the universe. We can only establish correlations. Correlations between events that occur in our universe is the furthest "truth" we can establish about the universe short of a full on proof. What this means is that nothing in the physical universe can be proven. Proof is the domain of maths and logic, correlations is the domain of science. Science cannot prove anything, it can only establish correlations and causations. The reason this occurs is because at any time in the future one can observe an event that contradicts a hypothesis. You can hypothesize that all birds have wings and observe 2 trillion birds with wings but you never know when one day you'll observe a bird without wings disproving your entire hypothesis. That is why nothing can be proven, you can only correlate things through observation. The other interesting part about correlation is what it isn't: Causation. People often talk about how correlation is not causation but people never talk about what causation is and how to establish it. If I can't empirically use correlation to establish causation how on earth is causation ever formally established? People rarely question this disconnect. The fact is, causation is rarely formally established but a method does exist and it's subtle. If I observe that whenever Bob flicks a switch the light comes on then I established that the light coming on is correlated with Bob flipping the switch. This is as far as I can go with just observation. To establish causation I must make myself both an observer and an entity that is part of the system itself. I have to take control and flip the switch randomly and observe that when I don't flip the switch nothing happens and when I do flip the switch the lights come on. By doing this I establish causation. To establish causation to higher and higher degrees I need to Cause (keyword) random events and make sure that a cause influences an effect AND absence of a cause and therefore absence of an effect occurs. Also note that establishing causation is not proof. At any point in time in the future I can flip the switch and the light may not come on which is contradictory evidence for causation. Causation in the statistical sense is like correlation, you establish it to a degree of confidence but you can never Prove that A caused B.
- DavidSJ 6y agoNote that there are ways to test causal hypotheses without intervening. For example, suppose we wish to test the hypothesis that smoking causes lung cancer via the main mechanism of tar buildup in the lungs, against the alternative hypothesis that smoking is correlated with lung cancer because of a gene that predisposes people to both smoking and lung cancer (this example comes from Judea Pearl’s Book of Why). If the former hypothesis is true, then we should see a correlation between smoking behavior and tar deposits, and we should also see a correlation between tar deposits and lung cancer even after controlling for smoking behavior. Composing the causal effects at each stage, we can then calculate the indirect causal effect of smoking on lung cancer. If this suffixes to explain the correlation, then that rules out the alternative genetic explanation for the correlation. Of course, we can always propose ad-hoc further hypotheses that complicate the analysis: maybe the correlation between smoking and tar deposits is itself non-causative, etc. This only goes to show that scientific inquiry must be done with judgment and with expert domain knowledge, testing plausible hypotheses in good faith. But that’s true for detecting mere correlations too — we can always doubt our instruments, or claim the data is a statistical fluke. And it’s true in randomized controlled trials: it’s conceivable we didn’t properly randomize, etc.
- deleted 6y ago[deleted]
- rmrfstar 6y agoThis is like... intro-to-intro statistics, summarized in the last bit of the article: "yes, corr is like a rescaled regression coefficient." y = x*b + e b = cov(x,y)/var(x) = corr(x,y) * (std(y)/std(x)) It's too bad the article did not mention omitted variable bias. The two principal sources of spurious correlation are (1) measurement error, (2) omitted variable bias. It is much easier to grok omitted variable bias in the regression context, then pull it back to correlation with the scaling rule.
- monoideism 6y agoI last studied this stuff 20 years ago. What's the difference between ommitted variable and a confounding variable? Seems like a similar idea. Is it that an ommitted variable isn't part of a causal chain of an included variable, and a confounder is?
- rmrfstar 6y agoConfounding variable is a more confusing, less precise name for the same phenomenon. Omitted variable bias tells it like it is. A bias in a regression coefficient that results from an incorrectly specified model.
- monoideism 6y agoThanks. Assuming they refer to the exact same phenomenon, I agree that ommitted variable bias is better terminology. I wonder why Wikipedia has separate entries for them: https://en.wikipedia.org/wiki/Omitted-variable_bias https://en.wikipedia.org/wiki/Omitted-variable_bias https://en.wikipedia.org/wiki/Confounding https://en.wikipedia.org/wiki/Confounding
- rmrfstar 6y agoContext. The picture at the top of the confounding article gives it away. Those kinds of diagrams are common in "hierarchical Bayesian models" like LDA. In the simple linear setting, they are the same thing. Teaching people the "confounding variable" concept in a general setting before teaching them about "omitted variable" in a linear setting is like teaching people about Riemannian manifolds before teaching them about vector spaces. Correlation isn't a particularly useful concept outside of simple linear models.
- mxcrossb 6y agoI don’t understand any of the responses I’m reading. The question isn’t asking for an intuitive understanding of correlation. He’s just complaining that only normalized values are presented.
- deleted 6y ago[deleted]
- m463 6y agohttps://xkcd.com/552/ https://xkcd.com/552/
- krueger71 6y agoThe absurd stuff http://www.tylervigen.com/spurious-correlations http://www.tylervigen.com/spurious-correlations