3 ms·

Now that we've all said "feature selection is not dimensionality reduction" to our hearts' content, could we return to the point of the article? Regardless of

Now that we've all said "feature selection is not dimensionality reduction" to our hearts' content, could we return to the point of the article? Regardless of whether you're doing feature selection or dimensionality reduction, the point remains that, if you're doing supervised learning, PCA is just compressing your X space, without any regard to your y. It could be that the last principal component of X, containing only 0.1% of the variance, contains 100% of the correlation between X and y. Using PCA for dimensionality reduction in a supervised learning context means throwing out an unknown amount of signal, which could be up to 100% of the signal. Now for unsupervised, exploratory analysis, PCA is definitely a candidate, but there are plenty of often-better alternatives there too.

jstx1 5y ago
For me PCA is strictly in a bucket labeled "this comes up a lot in data science interviews but you never actually need to use it at work".
- civilized 5y ago
  For me the contents of this bucket are things like SVM, Naive Bayes, and k-means. PCA is actually in a slightly higher tier than those three.
  - kleene_op 5y ago
    k-means are actually useful for bootstrapping statistical mixtures when you don't have good priors. Just for this convenient use alone I would place them above PCA.
    - civilized 5y ago
      What does bootstrapping statistical mixtures mean? Why not just bootstrap in the regular old non-parametric way? Why inject k-means into it?
      - kleene_op 5y ago
        When you try to approximate your data with a mixture of distributions, say Gaussians for instance, you have to initialize their parameters with some values before running your EM loop. Finding good ones can be very problematic. The k-means process is a reasonable method to get good enough starting values without having to think/compute too much. https://en.wikipedia.org/wiki/Expectation%E2%80%93maximization_algorithm https://en.wikipedia.org/wiki/Expectation%E2%80%93maximizati...
        civilized 5y ago
        OK, so for you, "bootstrapping statistical mixtures" means "warm-starting EM mixture modeling". Personally I never would have guessed that, in retrospect it makes sense... to each their own!
- it_does_follow 5y ago
  You've never had to any kind of factor analysis in your work or done any searching for latent variables that map to customer/stakeholder question? Given the number of people I've worked with that are interested in modeling "engagement", I find this hard to believe. PCA is an incredibly valuable tool that I've used in most jobs I've had. It's just a terrible idea as a default part of a feature engineering pipeline (which is what the author is talking about in terms of "feature selection"), for reasons outline in this article. I suggest you don't be quite so quick to dismiss important concepts in this area, and before criticizing this post, at least read through it (I noticed your comment about misunderstand what the author is discussing by "feature selection" is the top comment here).
  - jstx1 5y ago
    > You've never had to Nope, never. I'm not dismissing PCA altogether; I'm sharing my experience and pointing out that some topics come up much more often in interviews than on the job.
    - nerdponx 5y ago
      My experience is the opposite of yours. It has never come up in a job interview but I have used it for real work several times.
hatmatrix 5y ago
If you're looking for the part of the signal in X that covaries with y, the first thing to try is partial least squares.
- mturmon 5y ago
  Yes. In the simplified linear setting we're considering... If you want to do dimension reduction and only have X, use PCA. If you want to do dimension reduction and also have a "y", consider PLS or CCA (canonical correlation analysis). Some background: https://www.cs.cmu.edu/~tom/10701_sp11/slides/CCA_tutorial.pdf https://www.cs.cmu.edu/~tom/10701_sp11/slides/CCA_tutorial.p...
ylks 5y ago
You nailed it.

jstx1 5y ago

For me PCA is strictly in a bucket labeled "this comes up a lot in data science interviews but you never actually need to use it at work".

civilized 5y ago
For me the contents of this bucket are things like SVM, Naive Bayes, and k-means. PCA is actually in a slightly higher tier than those three.
- kleene_op 5y ago
  k-means are actually useful for bootstrapping statistical mixtures when you don't have good priors. Just for this convenient use alone I would place them above PCA.
  - civilized 5y ago
    What does bootstrapping statistical mixtures mean? Why not just bootstrap in the regular old non-parametric way? Why inject k-means into it?
    - kleene_op 5y ago
      When you try to approximate your data with a mixture of distributions, say Gaussians for instance, you have to initialize their parameters with some values before running your EM loop. Finding good ones can be very problematic. The k-means process is a reasonable method to get good enough starting values without having to think/compute too much. https://en.wikipedia.org/wiki/Expectation%E2%80%93maximization_algorithm https://en.wikipedia.org/wiki/Expectation%E2%80%93maximizati...
      - civilized 5y ago
        OK, so for you, "bootstrapping statistical mixtures" means "warm-starting EM mixture modeling". Personally I never would have guessed that, in retrospect it makes sense... to each their own!
it_does_follow 5y ago
You've never had to any kind of factor analysis in your work or done any searching for latent variables that map to customer/stakeholder question? Given the number of people I've worked with that are interested in modeling "engagement", I find this hard to believe. PCA is an incredibly valuable tool that I've used in most jobs I've had. It's just a terrible idea as a default part of a feature engineering pipeline (which is what the author is talking about in terms of "feature selection"), for reasons outline in this article. I suggest you don't be quite so quick to dismiss important concepts in this area, and before criticizing this post, at least read through it (I noticed your comment about misunderstand what the author is discussing by "feature selection" is the top comment here).
- jstx1 5y ago
  > You've never had to Nope, never. I'm not dismissing PCA altogether; I'm sharing my experience and pointing out that some topics come up much more often in interviews than on the job.
  - nerdponx 5y ago
    My experience is the opposite of yours. It has never come up in a job interview but I have used it for real work several times.

hatmatrix 5y ago

If you're looking for the part of the signal in X that covaries with y, the first thing to try is partial least squares.

mturmon 5y ago
Yes. In the simplified linear setting we're considering... If you want to do dimension reduction and only have X, use PCA. If you want to do dimension reduction and also have a "y", consider PLS or CCA (canonical correlation analysis). Some background: https://www.cs.cmu.edu/~tom/10701_sp11/slides/CCA_tutorial.pdf https://www.cs.cmu.edu/~tom/10701_sp11/slides/CCA_tutorial.p...

ylks 5y ago

You nailed it.