4 ms·

The Spearman rank correlation of your two sets is 61% (after dropping cities that are not common to both). https://gist.github.com/hughdbrown/94016d0f477b855a8

The Spearman rank correlation of your two sets is 61% (after dropping cities that are not common to both). https://gist.github.com/hughdbrown/94016d0f477b855a893514131fc91f99 https://gist.github.com/hughdbrown/94016d0f477b855a893514131... Sadly, Salt Lake City is dropped from the set because of the missing foreign-born data.

jessriedel 10y ago
Cool, thanks! If we were to dramatically over-simplify by assuming a city's economic mobility is a simple sum of independent Gaussian variables, is the following correct? (1) Pearson and Spearman rank correlation are the same. (2) Percentage foreign born is more important than all other variables combined (since they would contribute 39% of the total variance)
- hughdbrown 10y ago
  > Pearson and Spearman rank correlation are the same I don't know what this means -- like you'd get the same value if you calculated the correlation by Pearson correlation and Spearman rank correlation? Or are you saying there is something called the Pearson rank correlation and asking how it relates to Spearman rank correlation? I have no knowledge of a Pearson rank correlation. I used Spearman rank correlation because it is applicable to this situation: it "assesses how well the relationship between two variables can be described using a monotonic function." [1] Pearson's correlation would be appropriate if I was looking for a linear relationship between two numerical values, which I don't have here. > Percentage foreign born is more important than all other variables combined (since they would contribute 39% of the total variance) Hmmm. I think that it is the r-squared that is the proportion of variance in Y that is accounted for by X. Accordingly, the Spearman correlation of 0.61105258893120507 means that 37.3% of the variance in Y is accounted for by X, so really there is 62.7% of the variance still to be accounted for by other variables. There could well be another variable that explains the relationship better. [2] [1] https://en.wikipedia.org/wiki/Spearman%27s_rank_correlation_coefficient https://en.wikipedia.org/wiki/Spearman%27s_rank_correlation_... [2] I am assuming here that the relationship of correlation to variance for Pearson correlation works the same as for Spearman. I think they should, since Spearman rank correlation is just Pearson correlation applied to pairs of ranks.
  - jessriedel 10y ago
    > I don't know what this means... I mean: under the stated assumption that the data is given by a sum of independent Gaussian variables, would the Pearson correlation of the Gaussian data be equal to the Spearman rank correlation of the corresponding ranked data. One can check from the definition that the answer is "yes". > Hmmm. I think that it is the r-squared that is the proportion of variance in Y that is accounted for by X. Yes, you're right, I needed to square this number. So, assuming Gaussian independent variables, 37% of the variance in mobility is explained by percentage foreign born. It would be pretty surprising if there were even an larger effect independent than this, so I'll self-grade my conjecture as "plausible".

jessriedel 10y ago

Cool, thanks! If we were to dramatically over-simplify by assuming a city's economic mobility is a simple sum of independent Gaussian variables, is the following correct? (1) Pearson and Spearman rank correlation are the same. (2) Percentage foreign born is more important than all other variables combined (since they would contribute 39% of the total variance)

hughdbrown 10y ago
> Pearson and Spearman rank correlation are the same I don't know what this means -- like you'd get the same value if you calculated the correlation by Pearson correlation and Spearman rank correlation? Or are you saying there is something called the Pearson rank correlation and asking how it relates to Spearman rank correlation? I have no knowledge of a Pearson rank correlation. I used Spearman rank correlation because it is applicable to this situation: it "assesses how well the relationship between two variables can be described using a monotonic function." [1] Pearson's correlation would be appropriate if I was looking for a linear relationship between two numerical values, which I don't have here. > Percentage foreign born is more important than all other variables combined (since they would contribute 39% of the total variance) Hmmm. I think that it is the r-squared that is the proportion of variance in Y that is accounted for by X. Accordingly, the Spearman correlation of 0.61105258893120507 means that 37.3% of the variance in Y is accounted for by X, so really there is 62.7% of the variance still to be accounted for by other variables. There could well be another variable that explains the relationship better. [2] [1] https://en.wikipedia.org/wiki/Spearman%27s_rank_correlation_coefficient https://en.wikipedia.org/wiki/Spearman%27s_rank_correlation_... [2] I am assuming here that the relationship of correlation to variance for Pearson correlation works the same as for Spearman. I think they should, since Spearman rank correlation is just Pearson correlation applied to pairs of ranks.
- jessriedel 10y ago
  > I don't know what this means... I mean: under the stated assumption that the data is given by a sum of independent Gaussian variables, would the Pearson correlation of the Gaussian data be equal to the Spearman rank correlation of the corresponding ranked data. One can check from the definition that the answer is "yes". > Hmmm. I think that it is the r-squared that is the proportion of variance in Y that is accounted for by X. Yes, you're right, I needed to square this number. So, assuming Gaussian independent variables, 37% of the variance in mobility is explained by percentage foreign born. It would be pretty surprising if there were even an larger effect independent than this, so I'll self-grade my conjecture as "plausible".