5 ms·
The thing with PCA for dimensionality reduction in regression problems, specifically, is that it carries an assumption: that small variations in your predictor
by sideshowb 11y ago
The thing with PCA for dimensionality reduction in regression problems, specifically, is that it carries an assumption: that small variations in your predictor variables will not account for large variation in your dependent variable. This may or may not be justified. If you regard small variations as noise then by all means throw them away; if you don't, then you risk throwing out a minor component X that has a major influence on Y.
Regularized regression such as ridge regression can achieve the same thing without the risk of throwing away small but significant variation. So I think a current trend is to replace PCA+regression with ridge regression. This paper seems to take that trend a step further by replacing PCA itself with ridge regression even when the desired outcome is PCA, which is neat.