14 ms·

Well-known paradox of R-squared is still buggin me

71 points by luu 2y ago

deleted 2y ago
[deleted]
lmm 2y ago
> The two states are much different in their politics! Are they? Sounds like they're both swing states, pretty close to 50-50, so which state you're from doesn't have a big effect on what your politics are likely to be. Which is exactly what the R^2 tells us. Where's the paradox?
- derbOac 2y ago
  That was my first thought as well. Also, traditional R squared with binary variables, or maybe categorical variables, never made much sense to me. The "meaning" of the variance I don't think is quite the same. You generally have to nonlinearly transform (e.g., logit) any linear model quantities into something else to put it on an observed variable scale.
- spenczar5 2y ago
  A ten point gap is not a swing state, that’s big. Its about the same as New Jersey vs Texas in the 2020 presidential vote.
  - zarzavat 2y ago
    In the US. In other countries 45/55 could easily be a swing state. For example the 2014 Scottish independence referendum was decided 45/55 and that is usually held to be a fairly close result, at least close enough that it hasn’t ended the question. So the real story here is voter polarization. The reason that Texas isn’t a swing state is because there’s a big hard core of immoveable voters at either end. So the actual population of swing voters is small.
    - Spivak 2y ago
      I'm not exactly sure I would call the current political climate in the US voter polarization, it's an odd framing for what is ostensibly party polarization. You have political conservatives and christian conservatives pulling the Republican party further to the right to the point where libertarians and capital-M Moderate Republicans like Romney, McCain, Fitzpatrick have more in common with Democrats than their own party which would sound insane to anyone in the 00's. I genuinely don't think that outside of a small minority who drank the Trump kool-aid the political views of the country at large have really changed. The biggest shift is that my friends who were and still are run-of-the-mill "Republicans" don't feel particularly represented right now, and I get it. With the murmurs of Biden maybe stepping down I think if the Democrats nominated a Moderate Republican they would win in a landslide. It's nuts how much I didn't appreciate having two genuinely good candidates in 2008/2012.
  - chipdart 2y ago
    > A ten point gap is not a swing state, that’s big. No, it isn't. Your "ten point gap" is something like a 5% delta. That's below the confidence level of some polls.
- DougBTX 2y ago
  > Where's the paradox? Exactly. I propose that the paradox is in first-past-the-post voting, a 5% swing leads to a 100% change in representation. How can that be?
  - gadders 2y ago
    Relax - they're all broken: https://www.investopedia.com/terms/a/arrows-impossibility-theorem.asp https://www.investopedia.com/terms/a/arrows-impossibility-th...
    - evan_piermont 2y ago
      Only if you reject the axiom of choice: https://www.jyotirmoy.net/posts/2013-10-28-arrow-ultrafilter.html https://www.jyotirmoy.net/posts/2013-10-28-arrow-ultrafilter...
      - lupire 2y ago
        Only if you have infinite voters.
    - xigoi 2y ago
      Arrow’s impossibility theorem only applies to ranked choice voting systems.
      - cubefox 2y ago
        ... which are almost never used in voting for governments.
        lupire 2y ago
        FPTP is a ranked choice system.
        cubefox 2y ago
        No it isn't.
ninjinxo 2y ago
If voters are split 60-40 on an issue, that doesn't mean that the odds are 60-40. You should instead be asking, what are the odds that that X voters could change their vote.
akira2501 2y ago
What paradox? People don't vote a particular way because they live in a state. The logic here would imply "Welp, I live in Kentucky, so I guess Red?" would be the expected mode at the voting booth.
- jncfhnb 2y ago
  Statistics does not require asserting causality.
  - akira2501 2y ago
    Yes but you do actually have to understand the outcome you're measuring for.
    - jncfhnb 2y ago
      Not to encounter the “paradox” of R2 being low while a real effect is identified
cperciva 2y ago
I don't see the problem. The R-squared is 0.01 for blue/red predicting individual votes, because both of the states in question are really just different shades of purple. The R-squared is 1.00 for predicting the total vote share and which party wins the state, because of course the red/blue binary completely determines those.
- energy123 2y ago
  R-squared is very small when the effect is small because it's squared, as the name implies. If he doesn't like that he should just use R by itself, which would turn that 0.01 into 0.1, and it'd turn that 0.16 into 0.4. R is the Pearson correlation coefficient of a univariable linear regression.
  - cubefox 2y ago
    (Who came up with R squared anyway? Seems like someone wanted to remove the possible minus sign from R but didn't know about the ABS function, so he used the square which has this side effect of making the numbers too small.)
    - carlob 2y ago
      I don't think it's about not knowing the abs function, more about the fact that the first derivative would be discontinuous and the second doesn't exist in 0. Variance had much nicer properties mathematically than absolute deviation.
      - mattalex 2y ago
        You can solve L1 regression using linear programming at fantastically large scales. In fact in many applications you do the opposite: go from squared to absolute because the latter fits into in lp
    - snarkconjecture 2y ago
      R squared has extremely nice properties. If you add a bunch of linearly uncorrelated variables together, then compute R² between each variable and the sum, the R² values will sum to 1.
c76 2y ago
Isn’t the phenomenon just related to the way the vote options are encoded? Use different methods and you will see different R^2 results. Aren’t the votes represented artificially on a continuous domain for the R^2 calculation but the actual values are categorical values?
- ayhanfuat 2y ago
  Not really. No matter how you encode extremes will be 0% and 100% and one option will be 45% and the other 55%.
parpfish 2y ago
part of this has to do with the fact that our intuitive sense of effect sizes don't really use proportions and we subconsciously start including sample sizes. If a state had an election with millions of voters and got a 55-45 result, it would be a decisive landslide victory; If a elementary school classroom had an election 20 voters had got a 55-45 split, it's be the narrowest possible margin of victory. Most would likely say that the effect in the former 'feels' much larger even though proportions are identical, which suggests that under the hood we're factoring in sample size to our intuition about effect sizes (probably something chi-square-ish). The result is that the framing of the problem can change our sense of how big the effects are. When we hear that these are state-level elections, we think it's a huge effect and feel that we should be able to do reverse inference. If it was reframed as an election on a much smaller sample, the paradox disappears and you'd say "of course you wouldn't be able to reverse that inference"
- thaumasiotes 2y ago
  This ties in to the difference between whether an effect is statistically significant ("does the effect exist?") and whether it's significant ("does the effect matter?"). It's very common to confuse the two ideas. In particular, in an election with many millions of votes and a 55-45 margin, it's common to describe the winner as receiving a mandate to rule, because it was so easy to determine who the winner was, despite the fact that they appear to be extremely unpopular. That's not a mandate in any ordinary sense.
  - fjkdlsjflkds 2y ago
    > whether an effect is statistically significant ("does the effect exist?") and whether it's significant ("does the effect matter?") The more accurate terms to describe this is whether an effect is significant (i.e., "do we have enough information to be able to claim it is different from zero?") vs. whether an effect is relevant (i.e., "is [our estimate of] the size of the effect meaningfully different from zero?"). Statistics can only address the first issue (effect significance); the second issue (effect relevance) requires domain knowledge beyond statistics (in some contexts, a difference of 0.01 units can be irrelevant, while it may be relevant in other contexts).
blt 2y ago
I'm no statistician, but the whole premise seems mismatched. Why are we using a tool from regression to analyze a classification problem?
- greesil 2y ago
  Yep. I'm also not a statistician, but linear regression that the blogger is using predicts the mean for each state, and this is being conflated with trying to predict p(color | state). The goodness of fit here would be better modeled cross entropy and not a standard deviation.
  - kgwgk 2y ago
    For what it's worth, the "blogger" is an statistician.
    - ayhanfuat 2y ago
      I mean, Andrew Gelman, he is quite the statistician.
      - greesil 2y ago
        Oh he only went to MIT, pfff. And wrote a textbook.
        Xcelerate 2y ago
        Haha, this is why I always Google whoever the author is to articles posted on HN before commenting. More than once I've thought "this person is an idiot" only to Google their name and find out they are a famous person in that field. Then I go back and re-read their article and realize I missed something more subtle going on.
        lupire 2y ago
        This seems to be a special case. The blog post was rebutted by Seth in the comments 2 weeks ago, same as Colin Percival's HN rebuttal, and Andrew didn't reply. It seems like a weird goof. Andrew was "buggin".
leto_ii 2y ago
As other commenters have pointed out in one way or another, the problem seems to actually be that this simplistic model of voter choice can't capture all the structure of the real world that humans can quickly infer from the setup. Things like: state elections have millions of voters, 55/45 is actually a decisive, not a narrow win etc. In a generic setup, imagine you have a binary classifier that outputs probabilities in the .45-.55 range - likely it won't be a really strong classifier. You would ideally like polarized predictions, not values around .5. Come to think of it, could this be an issue of non-ergodicity too ( hope I'm using the term right)? i.e. state level prior is not that informative wrt individual vote?
- jncfhnb 2y ago
  No, you want your model to be well calibrated. If the model accurately assessed a 0.55 probability of going blue, then that is what you want. People who try to correct for “unbalanced classes” and contort their model to give polarizing predictions are frankly being pretty dumb. The correct answer is to take your well calibrated probabilities and use you brain on what to do with them.
  - leto_ii 2y ago
    This is not a matter of class balance that much. If you want to predict which of two parties somebody will vote with, the most natural framing is that of binary classification. For that you need to threshold your predictions. Ideally you'd like your model to generate a bimodal distribution so that you can threshold without many false positives etc.
    - jncfhnb 2y ago
      Yes but the prompt here states that all we know is the probability is either 0.55 or 0.45. By definition this is the best model you can produce.
gpsx 2y ago
My statisitcs are a little rusty, so I might be off here. Someone correct me if I have this wrong. R^2 = 1 would be every voter in one state votes blue and every voter in the other votes red. R^2 = 0 would mean both states are exactly even between red and blue. The states are a lot closer to that. Again, my statistics are rusty so I'm no sure if this next part is valid, but sqaure root of .01 is .1, which doesn't seem like such a bad representation of the situation.
kazinator 2y ago
Nothing but endless cloudfare captchas here for me. Removing cookies for the domain doesn't help, because (doh) I've never visited it before.
dash2 2y ago
This comes up a lot in genetics. One crowd says "polygenic scores for education don't tell you much, because look how low the R-squared is!" Another crowd (including me) says "polygenic scores for education are a big deal, because look how big the effect size is!"
justk 2y ago
The math is correct, but I think the model used is not correct since it doesn't reflect that the variable s is dichotomous so rather a mixed model should be used. If we continue thinking that s is continuous we could think of this example: s=state is encoded as a continuous variable between -1 and 1 here people change state frequently and -1 reflects the person will vote in the blue state with probability 1 and s=1 that the person will vote in the red state with probability 1 while s=0 means that the person has the same probability of voting in the red or blue states. When s is near zero the model is not able to predict the preferences of the voter and this is the reason of the low predictive power of this model for a continuous s. The extreme cases s=-1 or s=1 could be rare for populations that move from one state to the other frequently so the initial intuition is misleaded to this paradox.
- mtts 2y ago
  This. R2 is not the correct measure to use. This article is a perfect example of the principle that simply doing math and getting results is not necessarily meaningful.
  - kgwgk 2y ago
    R² is a measure like any other. In this case it measures the relative reduction in MSE - which is low because the prediction of individual votes remains quite bad even if the state is taken into account. Does another measure give substantially different results?
    - deleted 2y ago
      [deleted]
    - justk 2y ago
      I think that you are using here a different definition of R^2 for example the way you are thinking of R^2 doesn't allow for an interpretation of the constant term used in the linear model for the formula of the R^2 to be true. What you are thinking is R^2 = 1 - mean(the variance in each state)/(total variance), but that is not the definition of R^2 for a linear model. As the user fskfsk.... says in another comment, here the constant term explains a lot of the variance so that the slope terms contains less information, that is not available using your definition or idea of R^2
jncfhnb 2y ago
R2 is more simply explained as the share of the error variance explained by the model out of the share of the error explained by the best guess, which is, in this case 0.5. Guessing 0.5 will have you wrong wrong by 0.5 100% of the time. SST is 25 for a 100 sample example. Guessing 0.55 for the 0.55 state will have you wrong by 0.45 55% of the time and 0.55 45% of the time for the other. SSE is 24.75 1- 24.75 / 25 = 0.01 Looking at it this way it’s not too hard to see why the R2 is bad. It barely explains any more difference in the individual behavior than the basic guess. R2 is not a great metric for percentages or classification problems like this.
- kgwgk 2y ago
  > Looking at it this way it’s not too hard to see why the R2 is bad. It barely explains any more difference in the individual behavior than the basic guess. Right. R² is 1% because the prediction is bad - only marginally better than the basic guess. > R2 is not a great metric for percentages or classification problems like this. Using a different metric won't improve the prediction. Is the Brier score a great metric for problems like this? The Brier score for the model is 0.2475. The Brier score for the "basic guess" is 0.25. The improvement in the Brier score for the model relative to the basic guess is 1%.
fosdad2131321 2y ago
There are two ways to resolve the paradox 1. if you insist on using the r-squared (i.e., a linear regression measure), then properly center and normalize your data, and model what you actually predict: the difference between the baseline (0.5) and the probability to vote for party 0 or party 1. If you model the outcomes as 0/1 without this, then you are using a model made for gaussian variables on what should be a logistic regression 2. if you can live with something that more accurately captures the idea of "explanatory power", you can use a GLM (logistic link function), do a logistic regression, and then use the log odds or another measure. In both cases, the variance explained by the state that you are in is 1, because of course it is, that's how the thought experiment is constructed - p(vote for party 1)=0.5+ \delta(state). "Paradoxes" like this are often interesting in the sense that they point to the math being the wrong math or you using it wrong, but instead people tend to assume that they are obviously understanding things correctly so it must be some weird property of the world (which then sometimes is used to construct some faulty conclusions as in some of the cited papers)
justk 2y ago
From (1) On the other hand, if the variation between the group means and the grand mean is small, and the variation within groups is large, this suggests there are no real differences in the group means i.e. the variations we observe is just sampling variation. The above is in the context of analysis of variance. In our example the means in each state are 0.55 and 0.45 and the total mean is 0.50 so first summand is small but the variances in the red and blue states are both 0.247, large summand, so the variations we observe are just sampling variations. Hence the state factor is not important and that explains the low R^2 value. Note that in each state the predicted value for the model is the group mean of that group. So analysis of variance explains that the OP result is not a paradox or something strange. https://saestatsteaching.tech/analysis-of-variance https://saestatsteaching.tech/analysis-of-variance
gweinberg 2y ago
The states (and even more so the sub-state regions) really are much more different than what you would think just looking at R vs D. A Democrat in a city the Democrats win 90-10 is likely a very different Democrat from one where they lose 60-40.
Chinjut 2y ago
If you think that's bad, the R^4 coefficient is even lower.

71 points by luu 2y ago

deleted 2y ago

[deleted]

lmm 2y ago

> The two states are much different in their politics! Are they? Sounds like they're both swing states, pretty close to 50-50, so which state you're from doesn't have a big effect on what your politics are likely to be. Which is exactly what the R^2 tells us. Where's the paradox?

derbOac 2y ago
That was my first thought as well. Also, traditional R squared with binary variables, or maybe categorical variables, never made much sense to me. The "meaning" of the variance I don't think is quite the same. You generally have to nonlinearly transform (e.g., logit) any linear model quantities into something else to put it on an observed variable scale.
spenczar5 2y ago
A ten point gap is not a swing state, that’s big. Its about the same as New Jersey vs Texas in the 2020 presidential vote.
- zarzavat 2y ago
  In the US. In other countries 45/55 could easily be a swing state. For example the 2014 Scottish independence referendum was decided 45/55 and that is usually held to be a fairly close result, at least close enough that it hasn’t ended the question. So the real story here is voter polarization. The reason that Texas isn’t a swing state is because there’s a big hard core of immoveable voters at either end. So the actual population of swing voters is small.
  - Spivak 2y ago
    I'm not exactly sure I would call the current political climate in the US voter polarization, it's an odd framing for what is ostensibly party polarization. You have political conservatives and christian conservatives pulling the Republican party further to the right to the point where libertarians and capital-M Moderate Republicans like Romney, McCain, Fitzpatrick have more in common with Democrats than their own party which would sound insane to anyone in the 00's. I genuinely don't think that outside of a small minority who drank the Trump kool-aid the political views of the country at large have really changed. The biggest shift is that my friends who were and still are run-of-the-mill "Republicans" don't feel particularly represented right now, and I get it. With the murmurs of Biden maybe stepping down I think if the Democrats nominated a Moderate Republican they would win in a landslide. It's nuts how much I didn't appreciate having two genuinely good candidates in 2008/2012.
- chipdart 2y ago
  > A ten point gap is not a swing state, that’s big. No, it isn't. Your "ten point gap" is something like a 5% delta. That's below the confidence level of some polls.
DougBTX 2y ago
> Where's the paradox? Exactly. I propose that the paradox is in first-past-the-post voting, a 5% swing leads to a 100% change in representation. How can that be?
- gadders 2y ago
  Relax - they're all broken: https://www.investopedia.com/terms/a/arrows-impossibility-theorem.asp https://www.investopedia.com/terms/a/arrows-impossibility-th...
  - evan_piermont 2y ago
    Only if you reject the axiom of choice: https://www.jyotirmoy.net/posts/2013-10-28-arrow-ultrafilter.html https://www.jyotirmoy.net/posts/2013-10-28-arrow-ultrafilter...
    - lupire 2y ago
      Only if you have infinite voters.
  - xigoi 2y ago
    Arrow’s impossibility theorem only applies to ranked choice voting systems.
    - cubefox 2y ago
      ... which are almost never used in voting for governments.
      - lupire 2y ago
        FPTP is a ranked choice system.
        cubefox 2y ago
        No it isn't.

ninjinxo 2y ago

If voters are split 60-40 on an issue, that doesn't mean that the odds are 60-40. You should instead be asking, what are the odds that that X voters could change their vote.

akira2501 2y ago

What paradox? People don't vote a particular way because they live in a state. The logic here would imply "Welp, I live in Kentucky, so I guess Red?" would be the expected mode at the voting booth.

jncfhnb 2y ago
Statistics does not require asserting causality.
- akira2501 2y ago
  Yes but you do actually have to understand the outcome you're measuring for.
  - jncfhnb 2y ago
    Not to encounter the “paradox” of R2 being low while a real effect is identified

cperciva 2y ago

I don't see the problem. The R-squared is 0.01 for blue/red predicting individual votes, because both of the states in question are really just different shades of purple. The R-squared is 1.00 for predicting the total vote share and which party wins the state, because of course the red/blue binary completely determines those.

energy123 2y ago
R-squared is very small when the effect is small because it's squared, as the name implies. If he doesn't like that he should just use R by itself, which would turn that 0.01 into 0.1, and it'd turn that 0.16 into 0.4. R is the Pearson correlation coefficient of a univariable linear regression.
- cubefox 2y ago
  (Who came up with R squared anyway? Seems like someone wanted to remove the possible minus sign from R but didn't know about the ABS function, so he used the square which has this side effect of making the numbers too small.)
  - carlob 2y ago
    I don't think it's about not knowing the abs function, more about the fact that the first derivative would be discontinuous and the second doesn't exist in 0. Variance had much nicer properties mathematically than absolute deviation.
    - mattalex 2y ago
      You can solve L1 regression using linear programming at fantastically large scales. In fact in many applications you do the opposite: go from squared to absolute because the latter fits into in lp
  - snarkconjecture 2y ago
    R squared has extremely nice properties. If you add a bunch of linearly uncorrelated variables together, then compute R² between each variable and the sum, the R² values will sum to 1.

c76 2y ago

Isn’t the phenomenon just related to the way the vote options are encoded? Use different methods and you will see different R^2 results. Aren’t the votes represented artificially on a continuous domain for the R^2 calculation but the actual values are categorical values?

ayhanfuat 2y ago
Not really. No matter how you encode extremes will be 0% and 100% and one option will be 45% and the other 55%.

parpfish 2y ago

part of this has to do with the fact that our intuitive sense of effect sizes don't really use proportions and we subconsciously start including sample sizes. If a state had an election with millions of voters and got a 55-45 result, it would be a decisive landslide victory; If a elementary school classroom had an election 20 voters had got a 55-45 split, it's be the narrowest possible margin of victory. Most would likely say that the effect in the former 'feels' much larger even though proportions are identical, which suggests that under the hood we're factoring in sample size to our intuition about effect sizes (probably something chi-square-ish). The result is that the framing of the problem can change our sense of how big the effects are. When we hear that these are state-level elections, we think it's a huge effect and feel that we should be able to do reverse inference. If it was reframed as an election on a much smaller sample, the paradox disappears and you'd say "of course you wouldn't be able to reverse that inference"

thaumasiotes 2y ago
This ties in to the difference between whether an effect is statistically significant ("does the effect exist?") and whether it's significant ("does the effect matter?"). It's very common to confuse the two ideas. In particular, in an election with many millions of votes and a 55-45 margin, it's common to describe the winner as receiving a mandate to rule, because it was so easy to determine who the winner was, despite the fact that they appear to be extremely unpopular. That's not a mandate in any ordinary sense.
- fjkdlsjflkds 2y ago
  > whether an effect is statistically significant ("does the effect exist?") and whether it's significant ("does the effect matter?") The more accurate terms to describe this is whether an effect is significant (i.e., "do we have enough information to be able to claim it is different from zero?") vs. whether an effect is relevant (i.e., "is [our estimate of] the size of the effect meaningfully different from zero?"). Statistics can only address the first issue (effect significance); the second issue (effect relevance) requires domain knowledge beyond statistics (in some contexts, a difference of 0.01 units can be irrelevant, while it may be relevant in other contexts).

blt 2y ago

I'm no statistician, but the whole premise seems mismatched. Why are we using a tool from regression to analyze a classification problem?

greesil 2y ago
Yep. I'm also not a statistician, but linear regression that the blogger is using predicts the mean for each state, and this is being conflated with trying to predict p(color | state). The goodness of fit here would be better modeled cross entropy and not a standard deviation.
- kgwgk 2y ago
  For what it's worth, the "blogger" is an statistician.
  - ayhanfuat 2y ago
    I mean, Andrew Gelman, he is quite the statistician.
    - greesil 2y ago
      Oh he only went to MIT, pfff. And wrote a textbook.
      - Xcelerate 2y ago
        Haha, this is why I always Google whoever the author is to articles posted on HN before commenting. More than once I've thought "this person is an idiot" only to Google their name and find out they are a famous person in that field. Then I go back and re-read their article and realize I missed something more subtle going on.
        lupire 2y ago
        This seems to be a special case. The blog post was rebutted by Seth in the comments 2 weeks ago, same as Colin Percival's HN rebuttal, and Andrew didn't reply. It seems like a weird goof. Andrew was "buggin".

leto_ii 2y ago

As other commenters have pointed out in one way or another, the problem seems to actually be that this simplistic model of voter choice can't capture all the structure of the real world that humans can quickly infer from the setup. Things like: state elections have millions of voters, 55/45 is actually a decisive, not a narrow win etc. In a generic setup, imagine you have a binary classifier that outputs probabilities in the .45-.55 range - likely it won't be a really strong classifier. You would ideally like polarized predictions, not values around .5. Come to think of it, could this be an issue of non-ergodicity too ( hope I'm using the term right)? i.e. state level prior is not that informative wrt individual vote?

jncfhnb 2y ago
No, you want your model to be well calibrated. If the model accurately assessed a 0.55 probability of going blue, then that is what you want. People who try to correct for “unbalanced classes” and contort their model to give polarizing predictions are frankly being pretty dumb. The correct answer is to take your well calibrated probabilities and use you brain on what to do with them.
- leto_ii 2y ago
  This is not a matter of class balance that much. If you want to predict which of two parties somebody will vote with, the most natural framing is that of binary classification. For that you need to threshold your predictions. Ideally you'd like your model to generate a bimodal distribution so that you can threshold without many false positives etc.
  - jncfhnb 2y ago
    Yes but the prompt here states that all we know is the probability is either 0.55 or 0.45. By definition this is the best model you can produce.

gpsx 2y ago

My statisitcs are a little rusty, so I might be off here. Someone correct me if I have this wrong. R^2 = 1 would be every voter in one state votes blue and every voter in the other votes red. R^2 = 0 would mean both states are exactly even between red and blue. The states are a lot closer to that. Again, my statistics are rusty so I'm no sure if this next part is valid, but sqaure root of .01 is .1, which doesn't seem like such a bad representation of the situation.

kazinator 2y ago

Nothing but endless cloudfare captchas here for me. Removing cookies for the domain doesn't help, because (doh) I've never visited it before.

dash2 2y ago

This comes up a lot in genetics. One crowd says "polygenic scores for education don't tell you much, because look how low the R-squared is!" Another crowd (including me) says "polygenic scores for education are a big deal, because look how big the effect size is!"

justk 2y ago

The math is correct, but I think the model used is not correct since it doesn't reflect that the variable s is dichotomous so rather a mixed model should be used. If we continue thinking that s is continuous we could think of this example: s=state is encoded as a continuous variable between -1 and 1 here people change state frequently and -1 reflects the person will vote in the blue state with probability 1 and s=1 that the person will vote in the red state with probability 1 while s=0 means that the person has the same probability of voting in the red or blue states. When s is near zero the model is not able to predict the preferences of the voter and this is the reason of the low predictive power of this model for a continuous s. The extreme cases s=-1 or s=1 could be rare for populations that move from one state to the other frequently so the initial intuition is misleaded to this paradox.

mtts 2y ago
This. R2 is not the correct measure to use. This article is a perfect example of the principle that simply doing math and getting results is not necessarily meaningful.
- kgwgk 2y ago
  R² is a measure like any other. In this case it measures the relative reduction in MSE - which is low because the prediction of individual votes remains quite bad even if the state is taken into account. Does another measure give substantially different results?
  - deleted 2y ago
    [deleted]
  - justk 2y ago
    I think that you are using here a different definition of R^2 for example the way you are thinking of R^2 doesn't allow for an interpretation of the constant term used in the linear model for the formula of the R^2 to be true. What you are thinking is R^2 = 1 - mean(the variance in each state)/(total variance), but that is not the definition of R^2 for a linear model. As the user fskfsk.... says in another comment, here the constant term explains a lot of the variance so that the slope terms contains less information, that is not available using your definition or idea of R^2

jncfhnb 2y ago

R2 is more simply explained as the share of the error variance explained by the model out of the share of the error explained by the best guess, which is, in this case 0.5. Guessing 0.5 will have you wrong wrong by 0.5 100% of the time. SST is 25 for a 100 sample example. Guessing 0.55 for the 0.55 state will have you wrong by 0.45 55% of the time and 0.55 45% of the time for the other. SSE is 24.75 1- 24.75 / 25 = 0.01 Looking at it this way it’s not too hard to see why the R2 is bad. It barely explains any more difference in the individual behavior than the basic guess. R2 is not a great metric for percentages or classification problems like this.

kgwgk 2y ago
> Looking at it this way it’s not too hard to see why the R2 is bad. It barely explains any more difference in the individual behavior than the basic guess. Right. R² is 1% because the prediction is bad - only marginally better than the basic guess. > R2 is not a great metric for percentages or classification problems like this. Using a different metric won't improve the prediction. Is the Brier score a great metric for problems like this? The Brier score for the model is 0.2475. The Brier score for the "basic guess" is 0.25. The improvement in the Brier score for the model relative to the basic guess is 1%.

fosdad2131321 2y ago

There are two ways to resolve the paradox 1. if you insist on using the r-squared (i.e., a linear regression measure), then properly center and normalize your data, and model what you actually predict: the difference between the baseline (0.5) and the probability to vote for party 0 or party 1. If you model the outcomes as 0/1 without this, then you are using a model made for gaussian variables on what should be a logistic regression 2. if you can live with something that more accurately captures the idea of "explanatory power", you can use a GLM (logistic link function), do a logistic regression, and then use the log odds or another measure. In both cases, the variance explained by the state that you are in is 1, because of course it is, that's how the thought experiment is constructed - p(vote for party 1)=0.5+ \delta(state). "Paradoxes" like this are often interesting in the sense that they point to the math being the wrong math or you using it wrong, but instead people tend to assume that they are obviously understanding things correctly so it must be some weird property of the world (which then sometimes is used to construct some faulty conclusions as in some of the cited papers)

justk 2y ago

From (1) On the other hand, if the variation between the group means and the grand mean is small, and the variation within groups is large, this suggests there are no real differences in the group means i.e. the variations we observe is just sampling variation. The above is in the context of analysis of variance. In our example the means in each state are 0.55 and 0.45 and the total mean is 0.50 so first summand is small but the variances in the red and blue states are both 0.247, large summand, so the variations we observe are just sampling variations. Hence the state factor is not important and that explains the low R^2 value. Note that in each state the predicted value for the model is the group mean of that group. So analysis of variance explains that the OP result is not a paradox or something strange. https://saestatsteaching.tech/analysis-of-variance https://saestatsteaching.tech/analysis-of-variance

gweinberg 2y ago

The states (and even more so the sub-state regions) really are much more different than what you would think just looking at R vs D. A Democrat in a city the Democrats win 90-10 is likely a very different Democrat from one where they lose 60-40.

Chinjut 2y ago

If you think that's bad, the R^4 coefficient is even lower.