3 ms·

Where does the the idea that PCA is optimal come from? I understand the second example (girlfriend taking a phone screen): some people never took statistics pa

Where does the the idea that PCA is optimal come from? I understand the second example (girlfriend taking a phone screen): some people never took statistics past a linear models course, so they use what they know. Understandable, if inefficient. But they don't seem to be claiming linear models are optimal all the time, they're just stuck in a local maximum of trying to speed up the system they have. But the PCA guy I really don't understand.

wenc 5y ago
PCA is optimal in that it returns orthogonal axes in directions of maximum covariance. It can be cast as an optimization problem. It is definitely not optimal in a classification sense.
prionassembly 5y ago
PCA is pretty close to optimal in situations where you want to preserve global structure and variance-covariance matrices are highly informative. This comes down to the min-max theorem for Rayleigh quotients; I won't grandstand going over the math here. It won't work well for gene sequences or text embeddings because the desired structure there is local (UMAP seems to reign supreme these days for that). An older explanation for PCA is that it's a sort of default factor model prior to rotations that introduce structural assumptions. (Varimax etc. factor analysis is really underrated in exploratory statistics; but then, by now data science training never introduces people to the FWL theorem, identification, etc. With large enough deep learning pretty much anything is possible -- also because using deep learning implies having truckloads of data -- but the median xgboost guy is way out of his element.)
- taylorius 5y ago
  "and variance-covariance matrices are highly informative" So - a distribution which is linear?
  - kurthr 5y ago
    I don't think that the distribution needs to be linear... simply the correlation between N elements needs to be linear (or 1st order dominant). It works great for estimating multiple (e.g. different causes) small deviations (over multiple elements) from a global response. If you're looking at those larger eigen values, then you're looking at the dominant mode shapes of the eigen vectors of correlation. As others have said, it's better for estimation (not for classification).
  - prionassembly 5y ago
    You keep using that word.
  - spekcular 5y ago
    A multivariate Gaussian, for example.

wenc 5y ago

PCA is optimal in that it returns orthogonal axes in directions of maximum covariance. It can be cast as an optimization problem. It is definitely not optimal in a classification sense.

prionassembly 5y ago

PCA is pretty close to optimal in situations where you want to preserve global structure and variance-covariance matrices are highly informative. This comes down to the min-max theorem for Rayleigh quotients; I won't grandstand going over the math here. It won't work well for gene sequences or text embeddings because the desired structure there is local (UMAP seems to reign supreme these days for that). An older explanation for PCA is that it's a sort of default factor model prior to rotations that introduce structural assumptions. (Varimax etc. factor analysis is really underrated in exploratory statistics; but then, by now data science training never introduces people to the FWL theorem, identification, etc. With large enough deep learning pretty much anything is possible -- also because using deep learning implies having truckloads of data -- but the median xgboost guy is way out of his element.)

taylorius 5y ago
"and variance-covariance matrices are highly informative" So - a distribution which is linear?
- kurthr 5y ago
  I don't think that the distribution needs to be linear... simply the correlation between N elements needs to be linear (or 1st order dominant). It works great for estimating multiple (e.g. different causes) small deviations (over multiple elements) from a global response. If you're looking at those larger eigen values, then you're looking at the dominant mode shapes of the eigen vectors of correlation. As others have said, it's better for estimation (not for classification).
- prionassembly 5y ago
  You keep using that word.
- spekcular 5y ago
  A multivariate Gaussian, for example.