3 ms·

The article's question: "Why are correlations meaningful?" Here's a simplified answer. Correlations are meaningful because they indicate how strongly variables

by jph 6y ago

The article's question: "Why are correlations meaningful?" Here's a simplified answer. Correlations are meaningful because they indicate how strongly variables can predict other variables, in some circumstances. This is valuable because some variables are easier to use than others. A positive correlation between two variables means they tend to move in the same direction (relative to their mean). A negative correlation between two variables means they tend to move in opposite directions. A strong correlation between two variables means that they tend to change in much the same kinds of relative magnitudes, and also that either variables can be used to generally predict the other. A weak correlation between two variables means that the two variables tend to change in somewhat the same kinds of relative magnitudes, sometimes with some kinds of exceptions, and also that either variable can be used to somewhat predict the other. Thus a strong positive correlation means that variables can predict each other effectively, and also move in the same direction. This is especially valuable for projects where one variable is easy to obtain, yet the other variable is hard to obtain. And it turns out that a strong positive correlation can be a good place to hunt for possible causality, such as hunting for something that causing both variables to change together. However, correlation doesn't imply causation. A strong positive correlation can be good evidence that a hypothesized causality is likely to be true. For example an experiment can test if X causes Y, by varying X, and discovering that X and Y tend to change together. Edit: the replies below are great-- thank you-- well worth reading. I'm editing my post to make it clearer that it's a simplified answer, and to add suggestions by the replies.

Galanwe 6y ago
I don't quite like this answer, because it simplifies a lot of things about correlations which are just not really accurate. > Corrleations are meaningful because they indicate how strongly variables can predict other variables. This is not wrong per se, but may be wrongly interpreted. 1) I think it is important to stress that the reverse is not true. If a correlation is 0, it does not mean the variables are independent. It means they are linearly independent. There are a lot of things out there which are very much linked but not with a linear relationship. 2) This is only true if the correlation was done properly. For instance, correlating non-stationary variables will spuriously yield a high correlation, giving a false impression of link. The classic example is trying to correlate stock prices between themselves. An other example of spurious correlation is when not enough points are given to get an accurate result. Looking at tstat, r2, etc. is as important as looking at the actual correlation coefficient. 3) You use the word "predict" here, which I don't quite like. Correlation can predict outcomes only in the case where you have proven that past information will continue to occur. This is not always the case. > A positive correlation between two variables means they tend to move in the same direction. This is misleading also. The variables tend to move in the same direction _relatively to their mean_! I can plot you very different looking curves that are highly correlated just because their mean vary differently. Especially because of potential outliers. > And it turns out that a strong positive correlation can be a good place to hunt for possible causality, such as hunting for something that causing both variables to change together. I tend to dislike that thought process. It is very tempting to infer causality from correlation. Especially when automating correlation at scale is affordable, so generating spurious correlation is easy. To me correlation is more a way to reinforce a hypothesis of causality that is already suspected. It won't give you the direction, but may hint you that you are on the right track. I mean, correlation should not be a clue to discover causality, as there are too many false positives. Correlation should be an intermediary step in studying a potential causality.
- bollu 6y ago
  Thank you for this. I learnt probability from Sheldon Ross. I now realise I don't grok correlations at all! I had the exact same mistakes in my mental model of correlation as OP. Where do I go (what do I read) to get a good account of correlation?
- monoideism 6y ago
  Good critiques, but I disagree with the last one. At least in practice in the sciences, correlation is frequently used to find good candidates for causation. In many scenarios, correlation is pretty much your only hope for finding a causal link between some variable and an outcome of interest.
  - monoideism 6y ago
    To clarify, I'm not suggesting that correlation implies causation here (indeed, I think they take away your programmer card if you can't quickly parrot the phrase, "correlation doesn't mean causation"). From the NIST Engineer's Handbook: "Generally, we first need to find and explore correlations and then try to establish causal relationships. It is much easier to find correlations as these are just properties of the data." So I'm saying that correlation is often used (and misused) in an attempt to find good candidates for causation. This is how much science and engineering is practiced. If you disagree with this, you disagree with some fundamentals of how the scientific method is used (ie, hypothesis forming). > To me correlation is more a way to reinforce a hypothesis of causality that is already suspected It certainly can be used in this way, but that doesn't mean it's the only scientifically-valid way to use it.
- TheOtherHobbes 6y ago
  This is a very good answer. Perhaps one thing to add is that the underlying problem is always a mistaken belief that you can draw a general conclusion from a limited data set. The data may be flawed in many ways, so there is no automatic assumption that any conclusion is correct. And - as this answer says - this is a completely different process to discovering a robust causal mechanism that can reliably explain a correlation and predict its persistence into the future.
ur-whale 6y ago
> And it turns out that a strong positive correlation can be a good place to hunt for possible causality I don't disagree, but I have yet to see a mathematically clean definition of "causality".
- lordnacho 6y ago
  Do you take issue with Pearl?
  - ur-whale 6y ago
    Didn't know about Pearl. Will dive in now, thank you. His intro to the topic bodes well: http://bayes.cs.ucla.edu/BOOK-2K/why.html http://bayes.cs.ucla.edu/BOOK-2K/why.html
- mensetmanusman 6y ago
  That’s interesting, because causality requires time for the entropy engine to move our reality forward. It’s not clear to me whether mathematical axioms would be outside of entropy driven time, or if the mere presence of operators in sequence requires entropy change to transmit the information and move the calculation forward.

by jph 6y ago

Galanwe 6y ago

I don't quite like this answer, because it simplifies a lot of things about correlations which are just not really accurate. > Corrleations are meaningful because they indicate how strongly variables can predict other variables. This is not wrong per se, but may be wrongly interpreted. 1) I think it is important to stress that the reverse is not true. If a correlation is 0, it does not mean the variables are independent. It means they are linearly independent. There are a lot of things out there which are very much linked but not with a linear relationship. 2) This is only true if the correlation was done properly. For instance, correlating non-stationary variables will spuriously yield a high correlation, giving a false impression of link. The classic example is trying to correlate stock prices between themselves. An other example of spurious correlation is when not enough points are given to get an accurate result. Looking at tstat, r2, etc. is as important as looking at the actual correlation coefficient. 3) You use the word "predict" here, which I don't quite like. Correlation can predict outcomes only in the case where you have proven that past information will continue to occur. This is not always the case. > A positive correlation between two variables means they tend to move in the same direction. This is misleading also. The variables tend to move in the same direction _relatively to their mean_! I can plot you very different looking curves that are highly correlated just because their mean vary differently. Especially because of potential outliers. > And it turns out that a strong positive correlation can be a good place to hunt for possible causality, such as hunting for something that causing both variables to change together. I tend to dislike that thought process. It is very tempting to infer causality from correlation. Especially when automating correlation at scale is affordable, so generating spurious correlation is easy. To me correlation is more a way to reinforce a hypothesis of causality that is already suspected. It won't give you the direction, but may hint you that you are on the right track. I mean, correlation should not be a clue to discover causality, as there are too many false positives. Correlation should be an intermediary step in studying a potential causality.

bollu 6y ago
Thank you for this. I learnt probability from Sheldon Ross. I now realise I don't grok correlations at all! I had the exact same mistakes in my mental model of correlation as OP. Where do I go (what do I read) to get a good account of correlation?
monoideism 6y ago
Good critiques, but I disagree with the last one. At least in practice in the sciences, correlation is frequently used to find good candidates for causation. In many scenarios, correlation is pretty much your only hope for finding a causal link between some variable and an outcome of interest.
- monoideism 6y ago
  To clarify, I'm not suggesting that correlation implies causation here (indeed, I think they take away your programmer card if you can't quickly parrot the phrase, "correlation doesn't mean causation"). From the NIST Engineer's Handbook: "Generally, we first need to find and explore correlations and then try to establish causal relationships. It is much easier to find correlations as these are just properties of the data." So I'm saying that correlation is often used (and misused) in an attempt to find good candidates for causation. This is how much science and engineering is practiced. If you disagree with this, you disagree with some fundamentals of how the scientific method is used (ie, hypothesis forming). > To me correlation is more a way to reinforce a hypothesis of causality that is already suspected It certainly can be used in this way, but that doesn't mean it's the only scientifically-valid way to use it.
TheOtherHobbes 6y ago
This is a very good answer. Perhaps one thing to add is that the underlying problem is always a mistaken belief that you can draw a general conclusion from a limited data set. The data may be flawed in many ways, so there is no automatic assumption that any conclusion is correct. And - as this answer says - this is a completely different process to discovering a robust causal mechanism that can reliably explain a correlation and predict its persistence into the future.

ur-whale 6y ago

> And it turns out that a strong positive correlation can be a good place to hunt for possible causality I don't disagree, but I have yet to see a mathematically clean definition of "causality".

lordnacho 6y ago
Do you take issue with Pearl?
- ur-whale 6y ago
  Didn't know about Pearl. Will dive in now, thank you. His intro to the topic bodes well: http://bayes.cs.ucla.edu/BOOK-2K/why.html http://bayes.cs.ucla.edu/BOOK-2K/why.html
mensetmanusman 6y ago
That’s interesting, because causality requires time for the entropy engine to move our reality forward. It’s not clear to me whether mathematical axioms would be outside of entropy driven time, or if the mere presence of operators in sequence requires entropy change to transmit the information and move the calculation forward.