13 ms·
A New Coefficient of Correlation
- malshe 5y agoThis article is now published in the Journal of the American Statistical Association [Volume 116, 2021 - Issue 536] https://doi.org/10.1080/01621459.2020.1758115 https://doi.org/10.1080/01621459.2020.1758115
- mjcohen 5y agoThat costs $47 (US). The archiv article is free.
- ateng 5y agoI think pointing out that it is published on a known journal is to display the fact that the paper is now peer-reviewed
- ekianjo 5y agoPeer reviews dont mean much anymore these days.
- malshe 5y agoPerhaps to you. In academia we take them seriously.
- Nevermark 5y agoDo you have a measure of low correlation to support that? Regardless, I expect reviews are pretty straight forward for a concept that stands on its own like this one.
- derbOac 5y agohttps://link.springer.com/article/10.1007/s11192-017-2516-6 https://link.springer.com/article/10.1007/s11192-017-2516-6
- sdoering 5y agoIn answer to sister comment by derbOac: Thanks for sharing this. On the other hand the cited paper clearly relates to interdisciplinary peer review. I am not sure how or if this can be transferred to mathematics.
- nl 5y agoIn Math they really do.
- malshe 5y agoYes, that’s right. I wanted to highlight that this is a peer-reviewed article published in one of the top statistical journals.
- neeleshs 5y agoThere is also another paper comparing the power of Chatterjee coefficient with other traditional ones. https://arxiv.org/abs/2008.11619 https://arxiv.org/abs/2008.11619
- abeppu 5y agoIt is interesting to note that the Chatterjee paper makes a point of mentioning Pearson, Spearman, Kendall's tau, whereas the ones focused on in this paper all appear as citations but aren't explicitly discussed.
- nestorD 5y agoTo be fair, Pearson, Spearman and Kendall's tau are the coeffisicent people use in practice. Had I been in the author's position I would have done the same: cite all the interesting developements but compare with what people actually use. Comparing with something people barely know should be nice but people have a limited attention span so I would push to that anexes at best and focus on the more important parts.
- johnfhan 5y agoFor Pearson, Spearman, and Kendall's tau: (i) neither one of them could detect complex (nonlinear/non-monotone) dependence; (ii) they are clearly very powerful in detecting linear/monotone dependence. All these results have been documented in textbooks and repeatedly talked in classes for over 50 years.
- ur-whale 5y agoIs there code?
- malshe 5y agoYes, the author has shared the link to R package here: https://cran.r-project.org/web/packages/XICOR/index.html https://cran.r-project.org/web/packages/XICOR/index.html Edit: R code from Dr. Chatterjee's Stanford page is here - https://souravchatterjee.su.domains//xi.R https://souravchatterjee.su.domains//xi.R If you have never worked with R, the code seems clunky so I suggest checking out Python implementation on Github here: https://github.com/czbiohub/xicor https://github.com/czbiohub/xicor The Python library is not from the original author though. But it's easy to read the code and it works with pandas as well.
- tpaschalis 5y agoIf anyone is interested, I've also published a Go implementation [1] of the code for float64 slices. Results seem to exactly match the R and Python implementation, so there will be a second pass focusing on performance, stability and support for categorical variables. [1] https://github.com/tpaschalis/xicor-go https://github.com/tpaschalis/xicor-go
- ur-whale 5y agoThanks, the Python code is very clear and simple and makes it super easy to understand the idea without having to digest the paper.
- zmachinaz 5y agoThe current version of the python lib seems to be extremely badly written code. Or is the algo so bad ? Takes something like 21s to compute the correlation for just 10k samples.
- flyingmutant 5y agoThis issue contains simple code that is claimed to be >300x faster: https://github.com/czbiohub/xicor/issues/17 https://github.com/czbiohub/xicor/issues/17
- fieldcny 5y agoDumb question, What is the significance of this?
- singhrac 5y agoWell, in the abstract it says: “[a coefficient] which is 0 if and only if the variables are independent and 1 if and only if one is a measurable function of the other”, the former property which is not true of general random variables (but is true of Gaussians, which is one part of the reason they are used everywhere). I’m not sure about the latter property, actually, but I also doubt it’s true. Worth noting the author is a highly regarded professor at Stanford.
- rwilson4 5y agoCorrelation typically means y is a linear function of x, but people usually interpet it (incorrectly) as: knowing x tells you something about y. If y = x^2, then y is determined completely by x, but since it's nonlinear the correlation may actually be zero depending on the distribution of x. This paper proposes a statistic that will indicate if y is related to any function of x, linear or nonlinear.
- KarlKemp 5y agoThis is... quite wrong? The dictionary says: 1. a mutual relationship or connection between two or more things 2. [Statistics] interdependence of variable quantities. 3. [Statistics] a quantity measuring the extent of the interdependence of variable quantities. The most sympathetic to your definition is Wikipedia: In statistics, correlation or dependence is any statistical relationship, whether causal or not, between two random variables or bivariate data. In the broadest sense correlation is any statistical association, though it actually refers to the degree to which a pair of variables are linearly related. And that's the mathematical formulation. Correlation also has a meaning in everyday speech, and mathematics doesn't have the authority to just adopt terms and then claim people are wrong after they've changed the meaning. Also correlation very definitely means that knowing <x> tells you something about <y>. And vice versa. Like, for example: its value. Or at least a better idea of it than pure guessing without correlation.
- The_rationalist 5y agoWhy woud someone use pearson or smearson coefficients vs this new one?
- rwilson4 5y agoIf the relationship is linear, I'm guessing a test based on Pearson has greater power.
- mellavora 5y agopower and interpretability. Assuming a linear relationship, if you know the correlation coefficient, you can predict unobserved values of y based on a known x with good accuracy. y = ax + b + error where strong correlation means error is small.
- im3w1l 5y agoSeems likely. The presented coefficient only looks at the ordering of the X-values, and how it relates to the ordering of the Y-values. All other information is thrown away. That's how it can be so general, but it should come at the expense of power.
- emerged 5y agoAh that makes sense intuitively. I was confused how non linear correlation could be detected without knowing anything about the function itself.
- tomrod 5y agoPearson is for affine/linear+intercept relationships. Spearman is for monotonic relationships.
- deleted 5y ago[deleted]
- The_rationalist 5y agocould this better represent semantic textual similarity? https://paperswithcode.com/sota/semantic-textual-similarity-on-sts-benchmark https://paperswithcode.com/sota/semantic-textual-similarity-...
- asdf_snar 5y agoI once attended a summer school in Saint-Flour, France, where Sourav Chatterjee gave a masterful exposition of results on large deviations for random graphs. All chalk talk, clear presentation. My impression is he's a deep thinker. On that basis alone I give such an article much more weight than the plethora of articles that pass before the eyes; reading the masters has always been a good rule of thumb.
- zaptheimpaler 5y agoHow is it possible to make a general coefficient of correlation that works for any non-linear relationship? Say if y=sha256(x), doesn't that mean y is a predictable function of x, but its statistically impossible to tell from looking at inputs/outputs alone?
- idealmedtech 5y agoPerhaps they mean continuous or differentiable functions (evident by the desire to put an order on the elements) edit: was wrong about sha256
- deleted 5y ago[deleted]
- hinoki 5y agoThe summary says that it works if it is a measurable function [0], which is structure preserving. So sha256 would scramble the input too much for it to be detected here. [0] https://en.m.wikipedia.org/wiki/Measurable_function https://en.m.wikipedia.org/wiki/Measurable_function
- kkylin 5y agoI've not yet read the article, just the abstract. But the abstract is pretty precise, and being "measurable" is a very weak constraint. For (computationally) complex functions like a hash, my first guess is the "escape clause" is in the number of samples needed for the statistic to converge to its expected value.
- m1sta_ 5y agoHow does this relate to Maximal Information Coefficient (MIC), MGC, PPS and the like?
- derbOac 5y agoThe paper discusses MIC in particular at least. They suggest that MIC sometimes overstates the strength of a noisy relationship, giving the example of a bivariate normal mixture (Figure 9). In that example MIC is 1, but the new measure is 0.48 or something, which seems more reasonable.
- derbOac 5y agoAlso worth looking at the cited and related paper with the same author: https://arxiv.org/abs/1910.12327 https://arxiv.org/abs/1910.12327 This follow-up paper presents a related measure of conditional dependence that has a "natural interpretation as a nonlinear generalization of the familiar partial R2 statistic for measuring conditional dependence by regression." The follow-up paper also provides some additional interpretive insights, I think. My intuitive impression is that both of these are important developments. I also have a very vague suspicion, based on the form of the function, that the correlation measure has some interpretation in terms of mutual information involving rank transformations of random variables.
- malshe 5y agoThanks for finding this article. I agree, these are important developments in particular because so many econometric models are now using machine learning without any distributional assumptions. Using correlation coefficients based on linearity is grossly insufficient.
- shahbazac 5y agoThis tweet contains a visualization comparison between linear correlation and “Chatarjee correlation:” https://twitter.com/adad8m/status/1474754752193830912?s=21 https://twitter.com/adad8m/status/1474754752193830912?s=21
- extremelearning 5y agoThis new coefficient of correlation is really really awesome, and this visualization shows its value in such a beautifully simple presentation. It would be great if someone who has Wikipedia edit privileges, can edit the Wikipedia article at [1] to describe/link how the Chatarjee's correlation coefficient solves many of the known limitation of Pearson's correlation coefficient. ;) [1] https://en.wikipedia.org/wiki/Pearson_correlation_coefficient https://en.wikipedia.org/wiki/Pearson_correlation_coefficien... (especially second top-left diagram)
- querez 5y agoEasy there. New correlation coefficients get proposed all the time (eg. the introduction of the linked paper lists ~10-20 alone!). It's not a good idea to add every newly proposed coefficient to established wiki pages, just because they trend on social media. Yes, the paper looks nice, but if you read any new paper proposing a new measure, they all do! They're meant to be written that way. Let the community decide and test and discuss, and if in 10 years this new coefficient is well accepted and has proven itself, we can think about your proposed edit. Doing it before is putting the cart before the horse, and is a recipe for astroturfing.
- deleted 5y ago[deleted]
- Mathnerd314 5y agoThe right place to add it would be here AFAICT: https://en.wikipedia.org/wiki/Correlation#Other_measures_of_dependence_among_random_variables https://en.wikipedia.org/wiki/Correlation#Other_measures_of_.... The article on Pearson's shouldn't get sidetracked by discussing other types of variables. (or rather, it already has too many) But the part that one would add would not necessarily be the definition of the coefficient ξn, but rather the interesting discussion at the beginning about what makes for a good correlation coefficient.
- laplacesdemon48 5y agoIs there a way to extend this to measure a multivariate relationship? For example: Cor(X, Y & Z) I know you could run them pairwise but it’s possible Cor(X, Y) and Cor(X, Z) are close to zero but Cor(X, Y & Z) is close to 1.
- tomrod 5y agoThis is a good question. When I read the article I'll keep it in mind!
- kortex 5y agoSeems easy enough to play around with, and need not be strictly numbers either, as long as rank is defined on your fields (they are sortable...total ordering? I'm rusty on my terminology). Basically sort X, take the variance of the rank of Y, Z, etc. Much in the same way you would compute multi variable correlation.
- SubiculumCode 5y agoI think there could be a lot of use of distance based correlations https://towardsdatascience.com/introducing-distance-correlation-a-superior-correlation-metric-d569dc8900c7 https://towardsdatascience.com/introducing-distance-correlat...
- nextaccountic 5y agoCan information theoretic measures like Granger causality[0] be used for this purpose? [0] https://en.wikipedia.org/wiki/Granger_causality https://en.wikipedia.org/wiki/Granger_causality
- csee 5y agoGranger causality is about linear inter-temporal dependence. Chatterjee correlation is about non-linear contemporaneous (or time-independent) dependence. They're tools for quite different applications.
- pizza 5y agoLooks like you could do the opposite: use this measure as a proxy for correlation between input data X and Y, and seeing if increased correlation between X and Y is superior to just prediction of Y without X. Maybe model the correlation value going over a threshold as a sequence of Bernoulli trials
- _bz2r 5y agoThis is incredibly cool. Thanks HN for bringing things like this to the front page, which I would have never otherwise heard about.
- deleted 5y ago[deleted]
- kortex 5y agoThere are so many super cool aspects to this! - it's always fun when a new equation or formula is discovered, doubly so when it is very practical - actually really easy to wrap your head around it - seems very computationally efficient. Basically boils down to sorts and distance - not limited to strict numeric fields ( integers and reals). Anything with an ordering defined can act as your Xs/Ys: characters, words, complex/vectors (under magnitude). I think you could even apply it recursively to divide and conquer high dimensional datasets. - looks useful in both LTI and non stationary signal analysis - possible use cases: exoplanet search, cosmology in general, ecology, climate, cryptanalysis, medicine, neuroscience, and of course, someone will find a way to win (more likely lose) money on stonks.
- thesz 5y agoYou can update Pearson correlation online, you cannot update online OP correlation coefficient. Pearson and other correlation coefficients are linear, O(n), sorting would incur logiarithmic multiplier O(NlogN). Thus, it is not computationally efficient. To break ties randomly one has to have good random number generator, which is a problem in itself. Finally, if you want to have differentiable version of this correlation coefficient, you will need to use sorting networks which are O(Nlog^2(N)). But, it is a cool idea nevertheless, it brings up use of ranking in statistics and there are ties to other areas of the statistical science. For example, it appears that you can more efficiently prune language models if you use rank metric than probability [1]. [1] https://aclanthology.org/P02-1023.pdf https://aclanthology.org/P02-1023.pdf
- TeeMassive 5y ago> Pearson and other correlation coefficients are linear, O(n), sorting would incur logiarithmic multiplier O(NlogN). Thus, it is not computationally efficient. That doesn't seem a deal breaker to me. Sure if you're dealing with billions of data entries it will be 9 times slower, but throwing 9 times more computational power to solve a problem is far from something unheard of nowadays.
- tromp 5y ago
- lngnmn2 5y ago
- Jweb_Guru 5y agoI mean, if you choose a bullshit ordering relationship, sure. But that's (essentially) the only assumption. Ah, I missed the reference to "the humanities" the first time... seems like you have an ideological axe to grind.
- nl 5y agoWhat are you talking about? Correlation is absolutely useful for analysis of non-physical based systems.
- lngnmn2 5y ago"analysis"
- hdm41bc 5y agoOne of the aspects of math papers that I dislike is how unapproachable they are if you’re unfamiliar with some of the terminology and conventions. The esoteric symbols don’t make it any easier to Google their definitions either. For instance, what does this mean? > μ is the law of Y μ is usually the mean or average. Is “law” something else?
- ebb_earl_co 5y agoThe law of Y means the distribution of Y. See the last phrase from this [0] StackOverflow answer. [0] https://math.stackexchange.com/a/1397467 https://math.stackexchange.com/a/1397467
- hdm41bc 5y agoGreat thanks! I’ll remember to use math.stackexchange.com as a resource in the future.
- 331c8c71 5y agoDo you think a random production code snippet is approachable for someone lacking the necessary background? The notation in this paper is totally standard. EDIT: \mu there refers to a probability measure. Nothing to do with averages.
- hdm41bc 5y agoYeah, working code snippets would be great. That would provide an unambiguous implementation that someone could use to dig into the underlying functions used and learn the basic concepts that would be tedious for the author to go through. In terms of the notation, it seemed like the author actually tried to keep his paper accessible, so my complaint isn’t with the author. My gripe is more with math notation in general. In my opinion, unless you’ve read the appropriate textbooks or taken the right classes, math notation is hard to learn. The symbols are hard to Google for. Integral symbols, R for real numbers, sigma, delta, the round E that stands for IN are not found on a standard keyboard so it’s challenging for a layman to Google and learn that notation on their own. Math evolved over millennia and the notation wasn’t constructed with SEO in mind, so I understand why things are the way they are, but it’s a stumbling block for the uninitiated trying to learn more advanced math. Maybe there are resources like math.stackexchange out there that I’m unaware of that would help make learning notation more approachable.
- ndynan 5y agoMy understanding is that one of the major critiques of statistics, especially its use in psychology, has been the use of models which are derived from the mean. There are inherent flaws/assumptions to this approach which Peter Molenaar has done extensive work to critique (See Todd Rose's book on the subject). For anyone who understands the technique presented in this paper, does it also depend on the mean as a model like when calculating Pearson's r?
- extremelearning 5y agoThis is an order-based algorithm, so it is more related to the median than the mean. Another very useful consequence of being order-based, is that this new coefficient is much more robust to noise/outliers than the canonical correlation coefficient.
- LudwigNagasena 5y agoI think there are far bigger problems with the lack of theoretical foundations and abuse of p-values rather than with ergodicity or whatever is the pet peeve of Peter Molenaar.
- mellavora 5y agoIsn't Molenaar looking at networks of symptoms over time? Yes, in any multi-variate time series, when one is searching for relationships between the variables (and allowing that you may have multiple sets of time series which may have some type of grouping, i.e. observations from a set of people with one diagnosis vs observations from a set of people with a contrary or with no diagnosis), then yes, any attempt to find co-relations in the multivariate signal need to account for the underlying statistical distribution of the signal components. The normal distribution isn't a bad first a priori approximation, but you really need to check. side note- it also isn't clear that you can group by diagnosis, see, for example, https://pubmed.ncbi.nlm.nih.gov/29154565/ https://pubmed.ncbi.nlm.nih.gov/29154565/, which shows that even within diagnostic groups there is substantial individual variation.
- d883kd8 5y agoThere will be no AI fire alarm
- tdullien 5y agoI have a very naive question: What are the downsides of estimating mutual information instead? I have a math (but not statistics) background, and am sometimes bewildered by the many correlation coefficients that float around when MI describes pretty exactly what one wants ("how much does knowledge of one variable tell you about the other variable"?). So ... what am I not understanding?
- anewhnaccount2 5y agoI'm also interested in this, having tried and semi-successfully used mutual information for finding associations between multinomial variables. As an even more naive question, I find the actual selection of estimators bewildering. How do I know which estimator to use for mutual information? How do I know if my chosen estimator has converged or is doing a bad job on my data? Bringing it back to the topic at hand, does the estimator presented in the paper provide good estimates for a wider variety of cases than the mutual information plug-in estimator? If so I can see it might be nice for simplicity reasons alone. Can we have different estimators for this new correlation coefficient? Any ideas what that would look like?
- tbenst 5y agoMI is quite useful and widely used. It typically requires binning data though when distributions are unknown / empirically estimated. This approach is a rank-based score, more similar to Spearman correlation than Pearson. This allows for nonlinear relationships between the two variables. A slightly critical review on the work van be seen here: https://academic.oup.com/biomet/advance-article/doi/10.1093/biomet/asab028/6259083 https://academic.oup.com/biomet/advance-article/doi/10.1093/.... They argue that the older forms of rank correlation, namely D, R, and tau*, are superior. Nonetheless, it seems like a nice contribution to the stats literature, although I doubt the widespread use of correlation is going anywhere.
- spekcular 5y agoFreely available version of the review paper, for those who want it: https://arxiv.org/abs/2008.11619 https://arxiv.org/abs/2008.11619
- mellavora 5y agoBest line of the paper: "The formula is so simple that it is likely that there are many such coefficients, some of them possibly having better properties than the one presented below." suggesting that the author is seeking to highlight a basic principle, not tune for a specific application.
- infocollector 5y agoAm I reading the paper definition correctly? If yes, why does the vertical line and horizontal line in 2D have so different correlations? def corr(x,y): n = len(x) an_array = np.stack((x, y), axis=-1) sorted_array = an_array[np.argsort(an_array[:, 0])] r = rankdata(sorted_array[:,1]) s = sum(np.abs(np.diff(r))) return 1.0 - (3.0 \* s/(n\*n -1.0))
- yorwba 5y agoYou're using the formula for the case where there are no ties, but the vertical line has all X values tied and the horizontal line has all Y values tied. Also, the case where Y is a constant is explicitly excluded from consideration in the paper.
- nmca 5y agonon-pdf link to the article via arxiv vanity: https://www.arxiv-vanity.com/papers/1909.10140/ https://www.arxiv-vanity.com/papers/1909.10140/
- pmayrgundter 5y agoI didn't get the idea of ranking. But it's simple: "In statistics, ranking is the data transformation in which numerical or ordinal values are replaced by their rank when the data are sorted. For example, the numerical data 3.4, 5.1, 2.6, 7.3 are observed, the ranks of these data items would be 2, 3, 1 and 4 respectively. For example, the ordinal data hot, cold, warm would be replaced by 3, 1, 2." https://en.m.wikipedia.org/wiki/Ranking https://en.m.wikipedia.org/wiki/Ranking Also learned that a Spearman coeff is just the Pearson coefficient taken on the rank of the data, instead of on the raw data. But whereas Pearson/Spearman takes the sum of product of data/mean differences (Σ(x-x̄)(y-ȳ)/σxσy) where x̄ is mean and σ=std. dev., Chatterjee takes sum of rank differences (3Σ(rᵢ₊₁-rᵢ)/n²-1), concerning just the ranks of the Y data after the X,Y pairs have been sorted by X. But still missing the intuition for why the sum of difference of ranks is so useful or where the magic numbers come from.
- cornel_io 5y agojmount has an explanation elsewhere in this thread linking to https://win-vector.com/2021/12/26/how-to-read-sourav-chatterjees-basic-xicor-defenition/ https://win-vector.com/2021/12/26/how-to-read-sourav-chatter... which does a great job of explaining the intuition, but in a nutshell the normalization factor of 3 comes from the fact that if you select two random points between 0 and 1, the mean distance between will be 1/3 (which is pretty easy to write down and solve, boils down the the fact that a pyramid of height 1 that's 1x1 at the base has volume = 1/3).
- pmayrgundter 5y agoThanks! That did it
- jmount 5y agoI share some notes on reading/explaining the basic definition here: https://win-vector.com/2021/12/26/how-to-read-sourav-chatterjees-basic-xicor-defenition/ https://win-vector.com/2021/12/26/how-to-read-sourav-chatter...
- deleted 5y ago[deleted]
- neycoda 5y agoCorrelation never means causation, but it typically implies it, and often is directly entangled. Like the lead in car pumps being tied to city crime, the correlation likely was directly related to causation. That's not true of all correlation, but typically cause-effect has a correlation to something, even if there's only one and many other correlations that appear meaningful are meaningless.
- petters 5y agoI implemented this in Python and it seems to work really well! Thanks for this link! Pretty cool that it can also work for e.g. textual data in principle, as someone mentioned.
- credit_guy 5y agoThis is a bit of non-sense. If you have n pairs (x_i,y_i) with all x_i's and y_i's different, then you can always say that y is a function of x: you simply take a very wiggly function that passes through all the pairs. So this new "coefficient of correlation" should always return 100% in such cases. Of course, such a coefficient would be useless, and fortunately, this one doesn't do that. The point is however that in order for such a coefficient to not be trivially 100%, you need extra specifications, more precisely, you need to specify some type of desired smoothness (or tension) in the class of functions you are looking for. Let's make this more precise, and use R-squared (which this coefficient appears to be in some shape or form, since it is between 0 and 1, not -1 and 1). You look for relationships of the type y_i = f(x_i) + epsilon_i, where f is in some class of functions. Then this coefficient would be the ratio of the variance of f(x_i) and the variance of y_i. If you allow the class of functions to be arbitrary, there's no residual epsilon. So you need to specify somehow that class. In the case of the Pearson correlation, that class is the class of linear functions. If you want to generalize the Pearson correlation, you need to think of more general ways to specify the class of regression functions.