5 ms·
And the mistake is here : > Minitab calculates predicted R-squared by systematically removing each observation from the data set, estimating the regression equ
by waps 12y ago
And the mistake is here :
> Minitab calculates predicted R-squared by systematically removing each observation from the data set, estimating the regression equation, and determining how well the model predicts the removed observation. Like adjusted R-squared, predicted R-squared can be negative and it is always lower than R-squared.
(linked from the article, and better explained than anywhere in the article).
The mistake is in what is compared against what. Ideally a model would be independent from the data entirely. This is what you get if you correctly derive a model from first principles, and then verify that observed data points agree. The independence is perfect and it is valid to compare the two data sets. Any agreement builds confidence that the model is correct.
However that's not what's being done here. The "model" is a linear combination of the data set. So you're calculating the model generated from n data points with the model generated from n-1 data points, for each point. So you're effectively comparing the data against itself (which is also what R-squared is doing of course).
The correct way to go about this is to randomly divide your dataset in 2 pieces, ideally both should have a decent size. The second part you give to your colleague, with strict instructions to shoot you if you even ask to see that data. You build your model with your data points, and then ask your colleague to compare it with his data, and report only if they're close or not, NOT the values. If your fit is not close to his fit ("too much" - some fuziness here), you're overfitting.
The key difference here is the random part. The "adjusted predictive R squared" does not create independence between the compared data sets. This means it can't work correctly.
I realize this is the typical mathematician argument. I'm claiming here that there exists a way to make R squared and predictive R squared both be way too high, without giving any method to do so. But this is obviously wrong nonetheless.
- danieltillett 12y agoDo you really need a colleague here or just self-honesty once you have split the dataset?
- ajtulloch 12y ago> I realize this is the typical mathematician argument. Interestingly, a mathematician's argument on this exact method was actually given in 1977, in "An Asymptotic Equivalence of Choice of Model by Cross-Validation and Akaike's Criterion". [1] What Stone showed was that (modulo technicalities) performing this procedure - leave one out cross validation, the procedure you claim "cannot work correctly" - was asymptotically equivalent to minimising the AIC [2] of the model. So we have an asymptotic guarantee (for whatever that's worth) on the behaviour of this procedure. [1]: http://www.jstor.org/stable/2984877 http://www.jstor.org/stable/2984877 [2]: http://en.wikipedia.org/wiki/Akaike_information_criterion http://en.wikipedia.org/wiki/Akaike_information_criterion