5 ms·

practically speaking i have never had a reason to do this with PCA, but i find this reaction very weird. i have encountered the concept of using PCA to reduce t

• jstx1 5y ago
  My point is that dimensionality reduction != feature selection. PCA is dimensionality reduction, not feature selection.
  • bllguo 5y ago
    maybe the wording is not precise, but the meaning is clearly training on the top principal components instead of the whole data anyway in general i dont think the boundary between feature selection and dimensionality reduction is that sharp
    • etrautmann 5y ago
      It’s not, but the typical use for PCA as a dimensionality reduction step prior to a downstream classifier is to introduce a bottleneck to prevent overfitting. In some sense, that can be thought of as a means of avoiding specific feature selection by using an unsupervised method to discover the population covariance structure
      • ylks 5y ago
        That's precisely what the blog post deals with. In that PCA pre-processing step, nothing guarantees that principal components are better representations for your problem than original inputs; in fact PCA has nothing to do with your target, how could it guarantee principal components are better representation to predict it? Similarly, understanding the covariance structure of your original inputs will not necessarily help you predict your target better. Here's a simple example illustrating this. Take x a single feature highly informative about y, take z a (large) d-dimensional highly structured vector that is independent from both x and y. Now, consider using [x, z] to predict y. In this case, x happens to be a principal component as x and z are independent; it is associated to eigenvector [1, 0,...., 0] and eigenvalue Var(x). All other eigenvectors are of the form [0, u_1, ..., u_d] where [u_1, ..., u_d] is an eigenvector of Cov(z). All it would take for x to be the very last (i.e. 'least important') principal component is for Var(x) to be smaller than all eigenvalues of Cov(z), which is easily conceivable, irrespective of y! In your quest for a lower-dimensional 'bottleneck' using PCA you would end up removing x, the only useful feature for predicting y! This will certainly not reduce overfitting. PCA and other autoencoders work well as pre-processing step when there are structural reasons to believe low energy loss (or low reconstruction error) coincides with low signal loss. In tabular data, this tends to be the exception, not the norm.
  • j16sdiz 5y ago
    If you don't do any feature selection explicitly and use PCA dimensional reduction, you are letting PCA select the features
    • hervature 5y ago
      This is false though as your top eigenvectors are going to have non-zero entries in all components unless your original data has some features with 0 covariance which is highly unlikely. You still need to collect all the original features, you have not selected features that can be removed.
    • LudwigNagasena 5y ago
      That’s like saying that in a stacked model every layer is feature selection. That doesn’t really make sense. Feature selection is about which input data is used, not about how it is transformed during the whole prediction procedure.
  • deleted 5y ago
    [deleted]