3 ms·
I would chime in that cross-validation and its variants are extremely flexible in the sense they make very few assumptions on the model. No likelihood function
by srean 9y ago
I would chime in that cross-validation and its variants are extremely flexible in the sense they make very few assumptions on the model. No likelihood function ? no problem. Non-parametric no problem. Need non asymptotic numbers ? no problem. The model changed ? No problem again. One can use the exact same code and procedure.
AIC is controlled by the number of parameters and the quality of fit, but it is known in theory, as well as in practice that for many models, for example, neural networks, linear regression with additive iid Gaussian noise, the norm of the parameters matter more for prediction accuracy than the sheer number of parameters.
Not claiming that CV is the best model selection criteria, just listing out a few pluses for CV. The last word on model selection is yet to be spoken. It goes right to one of the core questions about model fitting and prediction.
- bendysnowflakes 9y agoYeah, I don't mean to make it sound like there's no good reason for CV... I came across as sort of harsh. I just feel like CV got taken up in application a lot because it makes intuitive sense but with relatively little in terms of theoretical scrutiny, especially relative to other alternatives. When it has been compared to alternatives, the comparisons are sort of weak in that they often involve strawman or unrepresentative assumptions. CV involves certain assumptions itself, for example, in terms of choices, that affect results in sort of unpredictable ways sometimes. I'm a big fan of bootstrapping too and have often felt that that avenue hasn't been explored as much as it could be.
- srean 9y agoOh your original comment wasn't harsh at all. I was just listing a few of CV's strengths which may have played a role in making it a popular choice