6 ms·

The only reason you need: PCA is not a feature selection algorithm Is the author misunderstanding something very basic or are they deliberately writing this wa

• bllguo 5y ago
  practically speaking i have never had a reason to do this with PCA, but i find this reaction very weird. i have encountered the concept of using PCA to reduce the dimensionality of your data before training many times, including in university. cursory searches show plenty of discussion, questions on crossvalidated, etc.
  • jstx1 5y ago
    My point is that dimensionality reduction != feature selection. PCA is dimensionality reduction, not feature selection.
    • bllguo 5y ago
      maybe the wording is not precise, but the meaning is clearly training on the top principal components instead of the whole data anyway in general i dont think the boundary between feature selection and dimensionality reduction is that sharp
      • etrautmann 5y ago
        It’s not, but the typical use for PCA as a dimensionality reduction step prior to a downstream classifier is to introduce a bottleneck to prevent overfitting. In some sense, that can be thought of as a means of avoiding specific feature selection by using an unsupervised method to discover the population covariance structure
        • ylks 5y ago
          That's precisely what the blog post deals with. In that PCA pre-processing step, nothing guarantees that principal components are better representations for your problem than original inputs; in fact PCA has nothing to do with your target, how could it guarantee principal components are better representation to predict it? Similarly, understanding the covariance structure of your original inputs will not necessarily help you predict your target better. Here's a simple example illustrating this. Take x a single feature highly informative about y, take z a (large) d-dimensional highly structured vector that is independent from both x and y. Now, consider using [x, z] to predict y. In this case, x happens to be a principal component as x and z are independent; it is associated to eigenvector [1, 0,...., 0] and eigenvalue Var(x). All other eigenvectors are of the form [0, u_1, ..., u_d] where [u_1, ..., u_d] is an eigenvector of Cov(z). All it would take for x to be the very last (i.e. 'least important') principal component is for Var(x) to be smaller than all eigenvalues of Cov(z), which is easily conceivable, irrespective of y! In your quest for a lower-dimensional 'bottleneck' using PCA you would end up removing x, the only useful feature for predicting y! This will certainly not reduce overfitting. PCA and other autoencoders work well as pre-processing step when there are structural reasons to believe low energy loss (or low reconstruction error) coincides with low signal loss. In tabular data, this tends to be the exception, not the norm.
    • j16sdiz 5y ago
      If you don't do any feature selection explicitly and use PCA dimensional reduction, you are letting PCA select the features
      • hervature 5y ago
        This is false though as your top eigenvectors are going to have non-zero entries in all components unless your original data has some features with 0 covariance which is highly unlikely. You still need to collect all the original features, you have not selected features that can be removed.
      • LudwigNagasena 5y ago
        That’s like saying that in a stacked model every layer is feature selection. That doesn’t really make sense. Feature selection is about which input data is used, not about how it is transformed during the whole prediction procedure.
    • deleted 5y ago
      [deleted]
• j16sdiz 5y ago
  PCA as feature selection is quite popular. Many text book on classification teach this.
  • rlayton2 5y ago
    So you have an example? I've never seen this and would like to see what people are doing here. I teach a lot of newbies data mining, so I'm very interested in how people get it wrong
    • bllguo 5y ago
      here's one: https://charlesreid1.github.io/circe/Digit%20Classification%20-%20PCA%20and%20SVC.html https://charlesreid1.github.io/circe/Digit%20Classification%...
      • hervature 5y ago
        But this is not so much "feature selection" as it is "compressing the data". It says right in the conclusion that the entire goal was "dimensionality reduction". In a very real way, PCA is selecting all features. That is, your data collection process remains unchanged. In a real feature selection, you would be able to say "ok, we don't need to collect X data anymore".
        • barrkel 5y ago
          The point of the article, though, is that dimensionality reduction which minimizes information loss (PCA) isn't necessarily dimensionality reduction which minimizes signal loss. A good example from the article is random features: random implies high information content but no signal value.
          • platz 5y ago
            exactly right, barrkel. Did you read the article, hervature?
      • 5y ago
• rlayton2 5y ago
  Yeah I thought I was reading the title wrong! For anyone that isn't aware, the role of pca is to create new (synthetic) features that represent the original features. It does not tell you which features of the original set are good for feature selection purposes.
  • pas 5y ago
    so what's a good way/algorithm/strategy/method/technique to select features? (obviously asking for a friend who's not that familiar with this, whereas I've all the black belts in feature engineering!)
    • z2210558 5y ago
      L1 regularisation is the usual way (see e.g. https://en.wikipedia.org/wiki/Lasso_(statistics) https://en.wikipedia.org/wiki/Lasso_(statistics))
      • microtonal 5y ago
        There are also some iterative methods like grafting (using feature gradients): https://www.jmlr.org/papers/volume3/perkins03a/perkins03a.pdf https://www.jmlr.org/papers/volume3/perkins03a/perkins03a.pd... and gain-based selection (using the improvement of the objective), see the appendix of: https://aclanthology.org/J96-1002.pdf https://aclanthology.org/J96-1002.pdf We used grafting for parser feature selection, for which it worked quite well: https://danieldk.eu/Research/Publications/ucnlg2011.pdf https://danieldk.eu/Research/Publications/ucnlg2011.pdf
      • clove 5y ago
        Psychologists use factor analysis.
      • ylks 5y ago
        Feature selection ought to be model-specific. Because a feature wasn't selected by Lasso (in a linear model) does not mean it cannot be useful in a non-linear model.
• pas 5y ago
  Could you elaborate on this a bit? Are there feature selection algorithms?
  • jstx1 5y ago
    PCA is a form for dimensionality reduction. It can be used to reduce your data to a only few features while preserving some properties like the predictive quality of your model. Simple example: you give PCA 10 input features, it gives back the principal components, you pick the first 2 principal components and use them in your regression model (you still select how many componenets to use yourself, PCA doesn't do that for you). Feature selection is well... selecting features. You start with your original 10 features and you pick which ones to use. You can do this based on domain knowledge, based on analysing variance, based on correlation between features, based on feature importance in some model, by adding/removing/shuffling features and seeing how your model performs etc.
    • Frost1x 5y ago
      >Feature selection is well... selecting features. You start with your original 10 features and you pick which ones to use. That sounds a lot like another form of dimensionality reduction to me. The part people trip up on is that dimensions have a very specific definition in the world of statistics. All of these are basically reductionist approaches to tackle representing more complex systems by shedding information, they just go about it in different ways.
      • jstx1 5y ago
        Good point, a look at the contents of the wikipedia page for dimensionality reduction makes it clear - https://en.wikipedia.org/wiki/Dimensionality_reduction https://en.wikipedia.org/wiki/Dimensionality_reduction Dimensionality reduction: 1. Feature selection. 2. Feature projection. 2.1 PCA So you could say that feature selection is dimensionality reduction but you can't say that PCA is feature selection.
        • 5y ago
• derbOac 5y ago
  I think you could think of PCA as a feature selection candidate when the prediction target is unknown or random?
• it_does_follow 5y ago
  It's very clear if you read the article that what the author is calling "feature selection" might be better termed "feature generation". He explicitly calls out what he means in the post: > When used for feature selection, data scientists typically regard z^p:=(z_1,…,z_p) as a feature vector than contains fewer and richer representations than the original input x for predicting a target y. I don't even think this is necessarily incorrect terminology, especially given the author's background of working primarily for Google and the like. It's the difference between considering feature section as "choosing from a list of the provided features" vs "choosing from the set of all possible features". The author's term makes perfect sense given the latter. PCA is used for this all the time in the field. There have been an astounding number of presentations I've seen where people start with PCA/SVD as the first round of feature transformation. I always ask "why are you doing that?" and the answer is always mumbling with shoulder shrugging. This is a solid post and I find it odd that you try to dismiss it as either ignorant or click bait, when a quick skim of it dismisses both of these options.