7 ms·

Why I’m Not a Fan of R-Squared

77 points by sndean 10y ago

tgb 10y ago
Had the author offered an alternative? Namely, can E^2 be calculated in practice?
- a_bonobo 10y ago
  I don't think you can calculate E^2 without the "true model", which you practically never have. The code uses the "true model" too: https://github.com/johnmyleswhite/r_squared/blob/master/utils.R#L63 https://github.com/johnmyleswhite/r_squared/blob/master/util... I guess the post is similar to Anscombe's quartet [1]: a warning not to blindly trust summary statistics. [1] https://en.wikipedia.org/wiki/Anscombe%27s_quartet https://en.wikipedia.org/wiki/Anscombe%27s_quartet
  - static_noise 10y ago
    So the solution the author proposes is both absolutely correct and absolutely useless in practice?
    - lottin 10y ago
      He's not proposing a solution as far as I can tell. He's simply using the E^2 statistic in order to illustrate the problem.
- yiyus 10y ago
  The author says that E2 is "generally unmeasurable". Indeed, the true model will be unknown in most cases. In my opinion, it is not about using E2 or other alternative measure to R2, but about being aware of the significance and validity of R2 and not blindly assuming that a higher value always means a better fit.
- data_scientist 10y ago
  My main issue with R^2 is that it is an artificial indicator. It has no business meaning. The best alternative is to build a metric that is related to your business issue. How many $ do you loss if you're wrong by 1%? Or by 10 units? Do your business loss is linear in your error? Is it symmetrical? These are the kind of questions you have to ask yourself. Defining a metric is hard, and there is no good shortcut.
Noseshine 10y ago
> “does my model perform worse than the true model?” What is "true model"? I can't make head nor tail of that term. I've never heard this before, nor does it make sense to me when I take just the word meaning.
- nabla9 10y ago
  True model is the probability distribution that generates the observed data.
  - DiabloD3 10y ago
    In other words, what was actually observed?
    - lottin 10y ago
      The stochastic process that generated the observed data.
    - irremediable 10y ago
      Not really. What is observed is data, not a model/distribution. One can fit a given model to this data, but they do not, on their own, tell you which model generated them.
    - enthdegree 10y ago
      Maybe you've gotten it but I don't know. Here is a maybe over-tired coin-flipping example: Say you have a coin that might be unfair, and you want to estimate its bias. You flip it a bunch of times, and it mostly lands heads. Predictions of the coin's bias based on this observed data are usually going to be that it's biased some % towards heads. (unless maybe you have a strong prior, but that's a different topic) But there is also a chance that really the coin is fair, or even biased towards tails and you just got unlucky in your flips. There's a mismatch between the true model (the coins actual bias) and the observed data (the result of your flips) because of this chance of `unluckiness'
    - nabla9 10y ago
acbart 10y ago
This is not very accessible for people with weak statistical backgrounds.
- yiyus 10y ago
  R-Squared is the most commonly used indicator of how good a model fits some data. The author discusses why this indicator can be misleading in some cases and shows an example. There is nothing interesting for people unfamiliar with R-Squared values or model fitting. But in fact, it is not complicated. If you feel curious, feel free to ask any specific question you have.
Rexxar 10y ago
That's seem an advantage for R²: if R² go down when you add more data, you know your model doesn't fit the data any more.
graeham 10y ago
Interesting article and I find it current for some problems I'm working on at the moment. I would add a few challanges. The example is a bit a of a strawman - a log(x) function has unique properties that make the Xmax-Xmin vs R^2 work like that. In real data, rarely does a single-variable 'true model' fit as well as the example either. Context is needed as well - depending on the use of the model, a linear or quadratic fit may be sufficient even for what is clearly a log dataset. The real failing on only for small values of x, maybe 5% of the range of total values. For this case, a bilinear model could fit quite well for the lower 5%, then the existing model for the upper 95%. It depends on the application. I like this phrase: "When deciding whether a model is useful, a high R2 can be undesirable and a low R2 can be desirable." Too often statistics are dominated by 'cutoff' values that people apply blindly to all situations. What do you think of robust regression methods, where obvious outliers are down-weighted?
- yummyfajitas 10y ago
  I'm not the author, but I'm a huge fan of robust regression. I make between $500-2000/month off a trading strategy based on such a method. (The method is basically Bayesian linear regression, but using an error model that has a heavier tail than a gaussian.) But a really important thing when using such methods is the lucas critique. When you need to use robust regression you are definitively in a space where all the simple and generic stuff (e.g. linear regression) doesn't work. So at this point it's important to validate the underlying assumptions behind the robust regression scheme. E.g., in my trading strategy, I've gone to great lengths to make sure the tail behavior I'm modelling is an overestimate of reality.
  - xoranth 10y ago
    Could you share more details about the robust regression you are using? All resources I could find online on robust regression would either point to Laplace distributed residuals, or some capped loss function.
kgwgk 10y ago
I find this very confusing, but I guess I'm not the intended target audience. Not that I say it's wrong, but I don't really see the point. Do people really expect R^2 to measure the fit of the model to the true model? R^2 measures the fit of the model to the data: i.e. how well does the model perform in predicting the outcomes. In his first example is clear that all the models are equally useless: the noise dominates and the predictive power of the models is close to zero. In the second example the predictive power of all the models has improved, because there is a clear trend. The true model predicts much better than the others now, but each model predicts better than in the previous example. In the first example, he concludes: "Even though R^2 suggests our model is not very good, E^2 tells us that our model is close to perfect over the range of x." Actually our model is "better than perfect". The R^2 for the linear model (0.0073) and for the quadratic model (0.0084) is slightly better than for the true model (0.0064). Of course this is not a problem specific to the R^2 measure (the MSE for the linear and quadratic fits is lower than for the true generating function) and can be explained because the linear and quadratic models overfit. E^2 is essentially the ratio the 1-R^2 values (minus one). We get -0.00083 and -0.00193 for the linear and quadratic models respectively (the ratios before substracting one are 0.9992 and 0.9981). In the second example,"visual inspection makes it clear that the linear model and quadratic models are both systematically inaccurate, but their values of R^2 have gone up substantially: R^2=0.760 for the linear model and R^2=0.997 for the true model. In contrast, E^2=85.582 for the linear model, indicating that this data set provides substantial evidence that the linear model is worse than the true model." The R^2 already indicates that the linear model (R^2=0.760) and the quadratic model (R^2=0.898) are worse than the true model (R^2=0.997). The fractions of unexplained variance are 0.240, 0.102 and 0.003 respecively and it's clear that the last one performs much better than the others before we take the ratios and substract one to calculate the E^2 values 85.6 and 35.7 for the linear and quadratic models respectively. (By the way: "we’ll work with an alternative R^2 calculation that ignores corrections for the number of regressors in a model." That's not an alternative R^2, that's the standard R^2. The adjusted R^2 that takes into account the number of regressors is the alternative one.)
- perturbation 10y ago
  I think that this example in particular is not the best for R^2. He's getting a really good fit for linear (especially when his first plot is centered in a narrow range), since log(x) has a nice Taylor expansions for log(x) ~ x - 1 in that region. For fits that are almost entirely close to the mean (no slope) I would expect to be saved by the F-test, but we're not here since there's a region where a linear fit fits the data at least somewhat well.
jostmey 10y ago
Learn about information theory. It is better to calculate the cross-entropy error or KL-divergence between the data and the model.
- chestervonwinch 10y ago
  Do you have any links where KL-divergence is used in the case of real-valued response variables? Why is it better?
otl1248 10y ago
This is a weird critique of R-Squared. You (should) learn very early on that comparing R-Squared values of different models is a no-no.
- achompas 10y ago
  And you're claiming it's weird because people learn this early on? Maybe that's where we disagree -- it's not as common to learn about the issues with R^2 as you might think.
  - otl1248 10y ago
    Maybe the fact that this is in the (reasonably brief) Wikipedia entry on R2 is some evidence? https://en.wikipedia.org/wiki/Coefficient_of_determination#Inflation_of_R2 https://en.wikipedia.org/wiki/Coefficient_of_determination#I... Also, I find that the OP is actually more confusing on this topic than Wikipedia. Not sure how else I can support that this is a basic fact about this metric. I'm not in the mood to find quotes in intro textbooks, etc.
scottfr 10y ago
I'm a huge fan of R^2 and you should be too. The simple way to think about R^2 is that it is a measure of relative predictive accuracy (and that is exactly how it is calculated). This is both a more accurate (and a more useful definition for most tasks the HN crowd would work on) than saying R^2 is a measure of the distance from the true model. All the figures and findings in the post are completely reasonable given that definition (the linear model performing better in the first case is due to the high variance of the author's data, other samplings would lead to the reverse result).
- mrandrewandrade 10y ago
  I think what John is saying is that people commonly use R^2 as measure of model fit where something like root mean squared error (RMSE) gives a better measure of model fit (by measuring the distance from the true model) depending on the model. Just using R^2 blindly for most tasks you would work on can lead to choosing an incorrect model. I think the main take-away from the post is to better understand the correct measure for model fit for your specific data. For example, if you are forecasting a time series with stationary demand, mean average deviation might be the best measure of model fit, but it the case where there is seasonality with a trend, the RMSE would be a better measure [1]. [1] http://robjhyndman.com/papers/foresight.pdf http://robjhyndman.com/papers/foresight.pdf
  - robzyb 10y ago
    > I think what John is saying is that people commonly use R^2 as measure of model fit where something like root mean squared error (RMSE) gives a better measure of model fit (by measuring the distance from the true model) I don't mean to be rude, but that is definitely not what he is saying. There are two important things I'd like to clarify: - It is wrong to call the alternative measure "the RMSE". The alternative that the article was proposing was a made-up measure called E^2 which measures the distance from the true model. - The author is not suggesting that people use E^2 instead of R^2 for any case. In fact, in almost all cases it is impossible to use E^2, because calculating it requires you to know the true model, and if you know the true model it would be very unlikely that you'd be wasting your time measuring other models. The author makes it clear that E^2 isn't really to be considered an alternative when he called it the "generally unmeasurable E^2".
sixbrx 10y ago
An interesting tangential argument I had with a colleague recently was how to actually calculate R^2 when testing model predictions on new data (emphasis on new data). He claimed the conventional way is as the square of the correlation coefficient of the two lists of data, whereas I was suggesting 1 - SSE/SST in accord with references I read (well, Wikipedia), which latter method yields lower values for his data. I know they are equal on training data when using linear regression with constant term, but generally they differ. Correlation way being scale and shift invariant to me disqualifies it as a measure for predicting new data. Unfortunately he was able to barrage me with many online references that just say that "R^2 is the square of the correlation coefficient", which without very careful reading of the context (fitting not predicting), and sometimes even with such careful reading, makes his interpretation look correct. I find the whole thing rather exasperating... It also occurs to me that just as a question of convention, that I may be wrong which wouldn't surprise me as I'm new to modeling. So: Do most modelers report correlation^2 as their R^2 values for holdout tests? I wonder have other modelers here encountered this confusion?
deleted 10y ago
[deleted]
mattb314 10y ago
"As such, R^2 answers the question: 'does my model perform better than a constant model?' But we often would like to answer a very different question: 'does my model perform worse than the true model?' " Maybe I'm over-generalizing here, but I think this fundamental assumption is untrue for most people who use statistical models to solve actual problems. All of the "undesirable" behavior of the R^2 metric makes complete sense when you view it as a comparison with the most naive model (the constant model), and while R^2 certainly doesn't measure how close to "true" a model is, it very well captures the utility of using a more sophisticated model over an extremely simple one, which (I believe) is a critical question. For example, if you had predict a process where measurement noise overwhelms variation in the process itself, as in his first example of log(x) from .99 to 1.0, then the constant model is pretty much the best you can do, and both log(x) itself and the linear model offer little additional benefit. Getting low R^2 values for those two models makes total sense--they offer no marginal benefit. In fact, if you had to make the decision "should I use log(x) or the constant model?", you're often better off going with the constant model for simplicity and predictability (unless you have domain knowledge motivating a different choice). I like well thought-out articles like this one on statistical concepts because they make me think about things I often take for granted, and while treating R^2 like a better version of RMSE or MAD is clearly wrong, it often better captures things people actually care about by taking the difficulty of the problem into account. If you're doing advanced statistical modeling, it's easy to underestimate how common it is for a beginner to celebrate getting (say) 96% classification accuracy on a 2-class problem where one class makes up more than 95% of the samples--an issue that using R^2 can quickly reveal. tldr: Awesome article but (imo) R^2 is more useful than most other metrics, not less

77 points by sndean 10y ago

tgb 10y ago

Had the author offered an alternative? Namely, can E^2 be calculated in practice?

a_bonobo 10y ago
I don't think you can calculate E^2 without the "true model", which you practically never have. The code uses the "true model" too: https://github.com/johnmyleswhite/r_squared/blob/master/utils.R#L63 https://github.com/johnmyleswhite/r_squared/blob/master/util... I guess the post is similar to Anscombe's quartet [1]: a warning not to blindly trust summary statistics. [1] https://en.wikipedia.org/wiki/Anscombe%27s_quartet https://en.wikipedia.org/wiki/Anscombe%27s_quartet
- static_noise 10y ago
  So the solution the author proposes is both absolutely correct and absolutely useless in practice?
  - lottin 10y ago
    He's not proposing a solution as far as I can tell. He's simply using the E^2 statistic in order to illustrate the problem.
yiyus 10y ago
The author says that E2 is "generally unmeasurable". Indeed, the true model will be unknown in most cases. In my opinion, it is not about using E2 or other alternative measure to R2, but about being aware of the significance and validity of R2 and not blindly assuming that a higher value always means a better fit.
data_scientist 10y ago
My main issue with R^2 is that it is an artificial indicator. It has no business meaning. The best alternative is to build a metric that is related to your business issue. How many $ do you loss if you're wrong by 1%? Or by 10 units? Do your business loss is linear in your error? Is it symmetrical? These are the kind of questions you have to ask yourself. Defining a metric is hard, and there is no good shortcut.

Noseshine 10y ago

> “does my model perform worse than the true model?” What is "true model"? I can't make head nor tail of that term. I've never heard this before, nor does it make sense to me when I take just the word meaning.

nabla9 10y ago
True model is the probability distribution that generates the observed data.
- DiabloD3 10y ago
  In other words, what was actually observed?
  - lottin 10y ago
    The stochastic process that generated the observed data.
  - irremediable 10y ago
    Not really. What is observed is data, not a model/distribution. One can fit a given model to this data, but they do not, on their own, tell you which model generated them.
  - enthdegree 10y ago
    Maybe you've gotten it but I don't know. Here is a maybe over-tired coin-flipping example: Say you have a coin that might be unfair, and you want to estimate its bias. You flip it a bunch of times, and it mostly lands heads. Predictions of the coin's bias based on this observed data are usually going to be that it's biased some % towards heads. (unless maybe you have a strong prior, but that's a different topic) But there is also a chance that really the coin is fair, or even biased towards tails and you just got unlucky in your flips. There's a mismatch between the true model (the coins actual bias) and the observed data (the result of your flips) because of this chance of `unluckiness'
  - nabla9 10y ago

acbart 10y ago

This is not very accessible for people with weak statistical backgrounds.

yiyus 10y ago
R-Squared is the most commonly used indicator of how good a model fits some data. The author discusses why this indicator can be misleading in some cases and shows an example. There is nothing interesting for people unfamiliar with R-Squared values or model fitting. But in fact, it is not complicated. If you feel curious, feel free to ask any specific question you have.

Rexxar 10y ago

That's seem an advantage for R²: if R² go down when you add more data, you know your model doesn't fit the data any more.

graeham 10y ago

Interesting article and I find it current for some problems I'm working on at the moment. I would add a few challanges. The example is a bit a of a strawman - a log(x) function has unique properties that make the Xmax-Xmin vs R^2 work like that. In real data, rarely does a single-variable 'true model' fit as well as the example either. Context is needed as well - depending on the use of the model, a linear or quadratic fit may be sufficient even for what is clearly a log dataset. The real failing on only for small values of x, maybe 5% of the range of total values. For this case, a bilinear model could fit quite well for the lower 5%, then the existing model for the upper 95%. It depends on the application. I like this phrase: "When deciding whether a model is useful, a high R2 can be undesirable and a low R2 can be desirable." Too often statistics are dominated by 'cutoff' values that people apply blindly to all situations. What do you think of robust regression methods, where obvious outliers are down-weighted?

yummyfajitas 10y ago
I'm not the author, but I'm a huge fan of robust regression. I make between $500-2000/month off a trading strategy based on such a method. (The method is basically Bayesian linear regression, but using an error model that has a heavier tail than a gaussian.) But a really important thing when using such methods is the lucas critique. When you need to use robust regression you are definitively in a space where all the simple and generic stuff (e.g. linear regression) doesn't work. So at this point it's important to validate the underlying assumptions behind the robust regression scheme. E.g., in my trading strategy, I've gone to great lengths to make sure the tail behavior I'm modelling is an overestimate of reality.
- xoranth 10y ago
  Could you share more details about the robust regression you are using? All resources I could find online on robust regression would either point to Laplace distributed residuals, or some capped loss function.

kgwgk 10y ago

I find this very confusing, but I guess I'm not the intended target audience. Not that I say it's wrong, but I don't really see the point. Do people really expect R^2 to measure the fit of the model to the true model? R^2 measures the fit of the model to the data: i.e. how well does the model perform in predicting the outcomes. In his first example is clear that all the models are equally useless: the noise dominates and the predictive power of the models is close to zero. In the second example the predictive power of all the models has improved, because there is a clear trend. The true model predicts much better than the others now, but each model predicts better than in the previous example. In the first example, he concludes: "Even though R^2 suggests our model is not very good, E^2 tells us that our model is close to perfect over the range of x." Actually our model is "better than perfect". The R^2 for the linear model (0.0073) and for the quadratic model (0.0084) is slightly better than for the true model (0.0064). Of course this is not a problem specific to the R^2 measure (the MSE for the linear and quadratic fits is lower than for the true generating function) and can be explained because the linear and quadratic models overfit. E^2 is essentially the ratio the 1-R^2 values (minus one). We get -0.00083 and -0.00193 for the linear and quadratic models respectively (the ratios before substracting one are 0.9992 and 0.9981). In the second example,"visual inspection makes it clear that the linear model and quadratic models are both systematically inaccurate, but their values of R^2 have gone up substantially: R^2=0.760 for the linear model and R^2=0.997 for the true model. In contrast, E^2=85.582 for the linear model, indicating that this data set provides substantial evidence that the linear model is worse than the true model." The R^2 already indicates that the linear model (R^2=0.760) and the quadratic model (R^2=0.898) are worse than the true model (R^2=0.997). The fractions of unexplained variance are 0.240, 0.102 and 0.003 respecively and it's clear that the last one performs much better than the others before we take the ratios and substract one to calculate the E^2 values 85.6 and 35.7 for the linear and quadratic models respectively. (By the way: "we’ll work with an alternative R^2 calculation that ignores corrections for the number of regressors in a model." That's not an alternative R^2, that's the standard R^2. The adjusted R^2 that takes into account the number of regressors is the alternative one.)

perturbation 10y ago
I think that this example in particular is not the best for R^2. He's getting a really good fit for linear (especially when his first plot is centered in a narrow range), since log(x) has a nice Taylor expansions for log(x) ~ x - 1 in that region. For fits that are almost entirely close to the mean (no slope) I would expect to be saved by the F-test, but we're not here since there's a region where a linear fit fits the data at least somewhat well.

jostmey 10y ago

Learn about information theory. It is better to calculate the cross-entropy error or KL-divergence between the data and the model.

chestervonwinch 10y ago
Do you have any links where KL-divergence is used in the case of real-valued response variables? Why is it better?

otl1248 10y ago

This is a weird critique of R-Squared. You (should) learn very early on that comparing R-Squared values of different models is a no-no.

achompas 10y ago
And you're claiming it's weird because people learn this early on? Maybe that's where we disagree -- it's not as common to learn about the issues with R^2 as you might think.
- otl1248 10y ago
  Maybe the fact that this is in the (reasonably brief) Wikipedia entry on R2 is some evidence? https://en.wikipedia.org/wiki/Coefficient_of_determination#Inflation_of_R2 https://en.wikipedia.org/wiki/Coefficient_of_determination#I... Also, I find that the OP is actually more confusing on this topic than Wikipedia. Not sure how else I can support that this is a basic fact about this metric. I'm not in the mood to find quotes in intro textbooks, etc.

scottfr 10y ago

I'm a huge fan of R^2 and you should be too. The simple way to think about R^2 is that it is a measure of relative predictive accuracy (and that is exactly how it is calculated). This is both a more accurate (and a more useful definition for most tasks the HN crowd would work on) than saying R^2 is a measure of the distance from the true model. All the figures and findings in the post are completely reasonable given that definition (the linear model performing better in the first case is due to the high variance of the author's data, other samplings would lead to the reverse result).

mrandrewandrade 10y ago
I think what John is saying is that people commonly use R^2 as measure of model fit where something like root mean squared error (RMSE) gives a better measure of model fit (by measuring the distance from the true model) depending on the model. Just using R^2 blindly for most tasks you would work on can lead to choosing an incorrect model. I think the main take-away from the post is to better understand the correct measure for model fit for your specific data. For example, if you are forecasting a time series with stationary demand, mean average deviation might be the best measure of model fit, but it the case where there is seasonality with a trend, the RMSE would be a better measure [1]. [1] http://robjhyndman.com/papers/foresight.pdf http://robjhyndman.com/papers/foresight.pdf
- robzyb 10y ago
  > I think what John is saying is that people commonly use R^2 as measure of model fit where something like root mean squared error (RMSE) gives a better measure of model fit (by measuring the distance from the true model) I don't mean to be rude, but that is definitely not what he is saying. There are two important things I'd like to clarify: - It is wrong to call the alternative measure "the RMSE". The alternative that the article was proposing was a made-up measure called E^2 which measures the distance from the true model. - The author is not suggesting that people use E^2 instead of R^2 for any case. In fact, in almost all cases it is impossible to use E^2, because calculating it requires you to know the true model, and if you know the true model it would be very unlikely that you'd be wasting your time measuring other models. The author makes it clear that E^2 isn't really to be considered an alternative when he called it the "generally unmeasurable E^2".

sixbrx 10y ago

An interesting tangential argument I had with a colleague recently was how to actually calculate R^2 when testing model predictions on new data (emphasis on new data). He claimed the conventional way is as the square of the correlation coefficient of the two lists of data, whereas I was suggesting 1 - SSE/SST in accord with references I read (well, Wikipedia), which latter method yields lower values for his data. I know they are equal on training data when using linear regression with constant term, but generally they differ. Correlation way being scale and shift invariant to me disqualifies it as a measure for predicting new data. Unfortunately he was able to barrage me with many online references that just say that "R^2 is the square of the correlation coefficient", which without very careful reading of the context (fitting not predicting), and sometimes even with such careful reading, makes his interpretation look correct. I find the whole thing rather exasperating... It also occurs to me that just as a question of convention, that I may be wrong which wouldn't surprise me as I'm new to modeling. So: Do most modelers report correlation^2 as their R^2 values for holdout tests? I wonder have other modelers here encountered this confusion?

deleted 10y ago

[deleted]

mattb314 10y ago

"As such, R^2 answers the question: 'does my model perform better than a constant model?' But we often would like to answer a very different question: 'does my model perform worse than the true model?' " Maybe I'm over-generalizing here, but I think this fundamental assumption is untrue for most people who use statistical models to solve actual problems. All of the "undesirable" behavior of the R^2 metric makes complete sense when you view it as a comparison with the most naive model (the constant model), and while R^2 certainly doesn't measure how close to "true" a model is, it very well captures the utility of using a more sophisticated model over an extremely simple one, which (I believe) is a critical question. For example, if you had predict a process where measurement noise overwhelms variation in the process itself, as in his first example of log(x) from .99 to 1.0, then the constant model is pretty much the best you can do, and both log(x) itself and the linear model offer little additional benefit. Getting low R^2 values for those two models makes total sense--they offer no marginal benefit. In fact, if you had to make the decision "should I use log(x) or the constant model?", you're often better off going with the constant model for simplicity and predictability (unless you have domain knowledge motivating a different choice). I like well thought-out articles like this one on statistical concepts because they make me think about things I often take for granted, and while treating R^2 like a better version of RMSE or MAD is clearly wrong, it often better captures things people actually care about by taking the difficulty of the problem into account. If you're doing advanced statistical modeling, it's easy to underestimate how common it is for a beginner to celebrate getting (say) 96% classification accuracy on a 2-class problem where one class makes up more than 95% of the samples--an issue that using R^2 can quickly reveal. tldr: Awesome article but (imo) R^2 is more useful than most other metrics, not less