4 ms·

I don't see the problem. The R-squared is 0.01 for blue/red predicting individual votes, because both of the states in question are really just different shade

I don't see the problem. The R-squared is 0.01 for blue/red predicting individual votes, because both of the states in question are really just different shades of purple. The R-squared is 1.00 for predicting the total vote share and which party wins the state, because of course the red/blue binary completely determines those.

energy123 2y ago
R-squared is very small when the effect is small because it's squared, as the name implies. If he doesn't like that he should just use R by itself, which would turn that 0.01 into 0.1, and it'd turn that 0.16 into 0.4. R is the Pearson correlation coefficient of a univariable linear regression.
- cubefox 2y ago
  (Who came up with R squared anyway? Seems like someone wanted to remove the possible minus sign from R but didn't know about the ABS function, so he used the square which has this side effect of making the numbers too small.)
  - carlob 2y ago
    I don't think it's about not knowing the abs function, more about the fact that the first derivative would be discontinuous and the second doesn't exist in 0. Variance had much nicer properties mathematically than absolute deviation.
    - mattalex 2y ago
      You can solve L1 regression using linear programming at fantastically large scales. In fact in many applications you do the opposite: go from squared to absolute because the latter fits into in lp
  - snarkconjecture 2y ago
    R squared has extremely nice properties. If you add a bunch of linearly uncorrelated variables together, then compute R² between each variable and the sum, the R² values will sum to 1.
    - cubefox 2y ago
      R (ρ) has also a nice property: > if X,Y,Z are "generic" random variables such that ρ(X,Y)=a and ρ(Y,Z)=b, we should on average expect that ρ(X,Z)=ab. https://www.lesswrong.com/posts/vfb5Seaaqzk5kzChb/when-is-correlation-transitive https://www.lesswrong.com/posts/vfb5Seaaqzk5kzChb/when-is-co...
  - justk 2y ago
    The reason of the apparent paradox: a) in this case the model is a mixed model b) second the variable are nominal so you have to select one of the pseudo R^2 models. For more information: (1) Pseudo R-squared: https://en.wikipedia.org/wiki/Pseudo-R-squared https://en.wikipedia.org/wiki/Pseudo-R-squared (2) R squared for mixed models – the easy way https://ecologyforacrowdedplanet.wordpress.com/2013/08/27/r-squared-in-mixed-models-the-easy-way/ https://ecologyforacrowdedplanet.wordpress.com/2013/08/27/r-... c) The R^2 used with a linear model requires a constant term, in this case the constant term or bias explains a lot about preferences (almost 50/50) so there is less information available for the slope term. Hope this helps.
    - vcdimension 2y ago
      @justk What you're talking about might make sense if there were more independent variables to consider, but in this case there's only one, state. So in fact you could say that there are two conditional linear models in the example; one for the first state (state=0), and one for the second (state=1). The model does the best job with the information available (state).
      - justk 2y ago
        Sorry, I edited my post several times and finally choose a short form with links other sources. If you fix state=1 then there are no more random variables so the R^2 doesn't have any meaning. Just for fun, what the model should predict for state = 0.5?, that corresponds to a person that is 50% in the red state and 50% in the blue state, I think a mixed model is appropriated here when the state variable is discrete, so that each value of the state variable represents a different part of the population, the other model should be used when people move a lot and change frequently the state where they vote in, but in that case you should have to consider the fluctuations in the total population in each state at the time of voting.
        vcdimension 2y ago
        @justk The R^2 value of 0.01 calculated on that webpage uses both states, not just one: the variance of the predicted values across both states is 0.55^3+0.45^3 - (0.55^2+0.45^2) ≃ 0.497 ≃ 0.5 I don't think it makes sense to use a mixed model in this case since the variance is the same for each state. A mixed model is used when the observations have some structured heteroskedasticity, i.e. different variances for different values of the independent variables.
  - kgwgk 2y ago
    Not sure about the "R squared" name, but the use of this quantity as "coefficient of determination" (the proportion of the variance in the dependent variable that is predictable from the independent variables) can be traced back to Sewall Wright's "Correlation and causation" (1921).
  - jncfhnb 2y ago
    When you’re making a linear regression you’re minimizing the sum of squared error which is 1:1 with R2. That is, you’re getting the best possible R2 achievable with a line. The reason we do that is because we are assuming the errors are normally distributed and finding the slope that gets the best possible R2 is equivalent to getting figuring out how to fit the line with the maximum likelihood estimator of the error (aka mean of the distribution). So ultimately it’s about curves. If you wanted to get a sense for why this is strongly desirable you should try to fit a linear regression using absolute error instead of squared error.
    - jvanderbot 2y ago
      Fun story: We once had a robot at a very big company you've all heard of that kept getting too close to walls when moving down hallway-like areas. The controls engineer swore he had tuned the controller to death and no further improvements could be had. A perception engineer took one look at the controller and saw linear error terms for distance left + distance right (distance to walls), changed it to distance left^2 + distance right^2, and the whole thing magically worked beautifully. Exercise for the reader: What position in the hallway minimizes sum of distances squared, vs what position(s) in the hallway minimize sum of distances without square. This is essentially the same problem you pose.
      - jncfhnb 2y ago
        Kind of funny but wouldn’t you want to just maximize(distance_left, distance_right) if you wanted the center? Edit: no, derp, just walked into the same problem lol. Maximize the min should work though
        jvanderbot 2y ago
        That has a unique solution, so it is infinitely better than the linear error, but is not a nice differentiable signal suitable for controls, I'd bet. But that wasn't in the problem statement.
        CamperBob2 2y ago
  - dxbydt 2y ago
    using just the ABS gives you the LAD. least absolute deviation is not used for a couple of reasons. LAD doesn’t have a unique solution - you can code it up and your program will give you a set of solutions which will all minimize the LAD. this looks like a feature, but for various pedagogical reasons, statistians insist on BLUE. LAD can give you the LUE ie linear unbiased estimator, but not the B ie best. also, LAD doesn’t have a closed form. so you can’t write an equation and say this is the LAD, then differentiate it and derive interesting properties about the estimator. since there’s no closed form, you can use numerical methods to find something that’s good enough like error below 1e-6 and declare that to be the LAD. but that will happen at a bunch of points so no uniqueness.
  - QuesnayJr 2y ago
    The idea of using the square is due to Carl Friedrich Gauss when calculating the orbit of Ceres. I suspect he was familiar with the idea of absolute value.
mihaaly 2y ago
> predicting individual votes Isn't that around the top of the statistics no-nos list? Probability theory in general? Predicting individual result based on the whole sample base? It was long ago and I am not in this field at all but my recollection tells me that it was mentioned in the beginning of the first class of probability theory 101.
- lupire 2y ago
  No, it's the opposite. The sample space measurement gives you the probability of an individual selected item. This is the fundamental reason why probabily works. But it requires uniform sampling.
kylebenzle 2y ago
Yes, you are right and statistics is confusing from the outside. My opinion is LLMs are just applied statistics and you see people losing their minds thinking the models have "come to life". Most people just really have no intuition for stats.
deleted 2y ago
[deleted]

energy123 2y ago

R-squared is very small when the effect is small because it's squared, as the name implies. If he doesn't like that he should just use R by itself, which would turn that 0.01 into 0.1, and it'd turn that 0.16 into 0.4. R is the Pearson correlation coefficient of a univariable linear regression.

cubefox 2y ago
(Who came up with R squared anyway? Seems like someone wanted to remove the possible minus sign from R but didn't know about the ABS function, so he used the square which has this side effect of making the numbers too small.)
- carlob 2y ago
  I don't think it's about not knowing the abs function, more about the fact that the first derivative would be discontinuous and the second doesn't exist in 0. Variance had much nicer properties mathematically than absolute deviation.
  - mattalex 2y ago
    You can solve L1 regression using linear programming at fantastically large scales. In fact in many applications you do the opposite: go from squared to absolute because the latter fits into in lp
- snarkconjecture 2y ago
  R squared has extremely nice properties. If you add a bunch of linearly uncorrelated variables together, then compute R² between each variable and the sum, the R² values will sum to 1.
  - cubefox 2y ago
    R (ρ) has also a nice property: > if X,Y,Z are "generic" random variables such that ρ(X,Y)=a and ρ(Y,Z)=b, we should on average expect that ρ(X,Z)=ab. https://www.lesswrong.com/posts/vfb5Seaaqzk5kzChb/when-is-correlation-transitive https://www.lesswrong.com/posts/vfb5Seaaqzk5kzChb/when-is-co...
- justk 2y ago
  The reason of the apparent paradox: a) in this case the model is a mixed model b) second the variable are nominal so you have to select one of the pseudo R^2 models. For more information: (1) Pseudo R-squared: https://en.wikipedia.org/wiki/Pseudo-R-squared https://en.wikipedia.org/wiki/Pseudo-R-squared (2) R squared for mixed models – the easy way https://ecologyforacrowdedplanet.wordpress.com/2013/08/27/r-squared-in-mixed-models-the-easy-way/ https://ecologyforacrowdedplanet.wordpress.com/2013/08/27/r-... c) The R^2 used with a linear model requires a constant term, in this case the constant term or bias explains a lot about preferences (almost 50/50) so there is less information available for the slope term. Hope this helps.
  - vcdimension 2y ago
    @justk What you're talking about might make sense if there were more independent variables to consider, but in this case there's only one, state. So in fact you could say that there are two conditional linear models in the example; one for the first state (state=0), and one for the second (state=1). The model does the best job with the information available (state).
    - justk 2y ago
      Sorry, I edited my post several times and finally choose a short form with links other sources. If you fix state=1 then there are no more random variables so the R^2 doesn't have any meaning. Just for fun, what the model should predict for state = 0.5?, that corresponds to a person that is 50% in the red state and 50% in the blue state, I think a mixed model is appropriated here when the state variable is discrete, so that each value of the state variable represents a different part of the population, the other model should be used when people move a lot and change frequently the state where they vote in, but in that case you should have to consider the fluctuations in the total population in each state at the time of voting.
      - vcdimension 2y ago
        @justk The R^2 value of 0.01 calculated on that webpage uses both states, not just one: the variance of the predicted values across both states is 0.55^3+0.45^3 - (0.55^2+0.45^2) ≃ 0.497 ≃ 0.5 I don't think it makes sense to use a mixed model in this case since the variance is the same for each state. A mixed model is used when the observations have some structured heteroskedasticity, i.e. different variances for different values of the independent variables.
- kgwgk 2y ago
  Not sure about the "R squared" name, but the use of this quantity as "coefficient of determination" (the proportion of the variance in the dependent variable that is predictable from the independent variables) can be traced back to Sewall Wright's "Correlation and causation" (1921).
- jncfhnb 2y ago
  When you’re making a linear regression you’re minimizing the sum of squared error which is 1:1 with R2. That is, you’re getting the best possible R2 achievable with a line. The reason we do that is because we are assuming the errors are normally distributed and finding the slope that gets the best possible R2 is equivalent to getting figuring out how to fit the line with the maximum likelihood estimator of the error (aka mean of the distribution). So ultimately it’s about curves. If you wanted to get a sense for why this is strongly desirable you should try to fit a linear regression using absolute error instead of squared error.
  - jvanderbot 2y ago
    Fun story: We once had a robot at a very big company you've all heard of that kept getting too close to walls when moving down hallway-like areas. The controls engineer swore he had tuned the controller to death and no further improvements could be had. A perception engineer took one look at the controller and saw linear error terms for distance left + distance right (distance to walls), changed it to distance left^2 + distance right^2, and the whole thing magically worked beautifully. Exercise for the reader: What position in the hallway minimizes sum of distances squared, vs what position(s) in the hallway minimize sum of distances without square. This is essentially the same problem you pose.
    - jncfhnb 2y ago
      Kind of funny but wouldn’t you want to just maximize(distance_left, distance_right) if you wanted the center? Edit: no, derp, just walked into the same problem lol. Maximize the min should work though
      - jvanderbot 2y ago
        That has a unique solution, so it is infinitely better than the linear error, but is not a nice differentiable signal suitable for controls, I'd bet. But that wasn't in the problem statement.
      - CamperBob2 2y ago
- dxbydt 2y ago
  using just the ABS gives you the LAD. least absolute deviation is not used for a couple of reasons. LAD doesn’t have a unique solution - you can code it up and your program will give you a set of solutions which will all minimize the LAD. this looks like a feature, but for various pedagogical reasons, statistians insist on BLUE. LAD can give you the LUE ie linear unbiased estimator, but not the B ie best. also, LAD doesn’t have a closed form. so you can’t write an equation and say this is the LAD, then differentiate it and derive interesting properties about the estimator. since there’s no closed form, you can use numerical methods to find something that’s good enough like error below 1e-6 and declare that to be the LAD. but that will happen at a bunch of points so no uniqueness.
- QuesnayJr 2y ago
  The idea of using the square is due to Carl Friedrich Gauss when calculating the orbit of Ceres. I suspect he was familiar with the idea of absolute value.

mihaaly 2y ago

> predicting individual votes Isn't that around the top of the statistics no-nos list? Probability theory in general? Predicting individual result based on the whole sample base? It was long ago and I am not in this field at all but my recollection tells me that it was mentioned in the beginning of the first class of probability theory 101.

lupire 2y ago
No, it's the opposite. The sample space measurement gives you the probability of an individual selected item. This is the fundamental reason why probabily works. But it requires uniform sampling.

kylebenzle 2y ago

Yes, you are right and statistics is confusing from the outside. My opinion is LLMs are just applied statistics and you see people losing their minds thinking the models have "come to life". Most people just really have no intuition for stats.

deleted 2y ago

[deleted]