PCA is pretty close to optimal in situations where you want to preserve global structure and variance-covariance matrices are highly informative. This comes dow

PCA is pretty close to optimal in situations where you want to preserve global structure and variance-covariance matrices are highly informative. This comes down to the min-max theorem for Rayleigh quotients; I won't grandstand going over the math here. It won't work well for gene sequences or text embeddings because the desired structure there is local (UMAP seems to reign supreme these days for that). An older explanation for PCA is that it's a sort of default factor model prior to rotations that introduce structural assumptions. (Varimax etc. factor analysis is really underrated in exploratory statistics; but then, by now data science training never introduces people to the FWL theorem, identification, etc. With large enough deep learning pretty much anything is possible -- also because using deep learning implies having truckloads of data -- but the median xgboost guy is way out of his element.)