3 ms·

PCA is a form for dimensionality reduction. It can be used to reduce your data to a only few features while preserving some properties like the predictive quali

by jstx1 5y ago

PCA is a form for dimensionality reduction. It can be used to reduce your data to a only few features while preserving some properties like the predictive quality of your model. Simple example: you give PCA 10 input features, it gives back the principal components, you pick the first 2 principal components and use them in your regression model (you still select how many componenets to use yourself, PCA doesn't do that for you). Feature selection is well... selecting features. You start with your original 10 features and you pick which ones to use. You can do this based on domain knowledge, based on analysing variance, based on correlation between features, based on feature importance in some model, by adding/removing/shuffling features and seeing how your model performs etc.

Frost1x 5y ago
>Feature selection is well... selecting features. You start with your original 10 features and you pick which ones to use. That sounds a lot like another form of dimensionality reduction to me. The part people trip up on is that dimensions have a very specific definition in the world of statistics. All of these are basically reductionist approaches to tackle representing more complex systems by shedding information, they just go about it in different ways.
- jstx1 5y ago
  Good point, a look at the contents of the wikipedia page for dimensionality reduction makes it clear - https://en.wikipedia.org/wiki/Dimensionality_reduction https://en.wikipedia.org/wiki/Dimensionality_reduction Dimensionality reduction: 1. Feature selection. 2. Feature projection. 2.1 PCA So you could say that feature selection is dimensionality reduction but you can't say that PCA is feature selection.
  - mywittyname 5y ago
    > but you can't say that PCA is feature selection. One of the steps in PCA is to select the top N eigenvectors sorted by eigenvalues. So while it is not feature selection, it can be argued that it involves a feature selection step.

by jstx1 5y ago

Frost1x 5y ago

>Feature selection is well... selecting features. You start with your original 10 features and you pick which ones to use. That sounds a lot like another form of dimensionality reduction to me. The part people trip up on is that dimensions have a very specific definition in the world of statistics. All of these are basically reductionist approaches to tackle representing more complex systems by shedding information, they just go about it in different ways.

jstx1 5y ago
Good point, a look at the contents of the wikipedia page for dimensionality reduction makes it clear - https://en.wikipedia.org/wiki/Dimensionality_reduction https://en.wikipedia.org/wiki/Dimensionality_reduction Dimensionality reduction: 1. Feature selection. 2. Feature projection. 2.1 PCA So you could say that feature selection is dimensionality reduction but you can't say that PCA is feature selection.
- mywittyname 5y ago
  > but you can't say that PCA is feature selection. One of the steps in PCA is to select the top N eigenvectors sorted by eigenvalues. So while it is not feature selection, it can be argued that it involves a feature selection step.