7 ms·
Principal Component Analysis Explained Visually
- strontian 5y agoWhat was used to make the visualizations?
- alexcnwy 5y agohttps://d3js.org/ https://d3js.org/ & https://threejs.org/ https://threejs.org/
- gentleman11 5y agoI bet you could use three.js as well
- gentleman11 5y ago> a transformation no different than finding a camera angle I’ve used PCA a bit in the past and it’s so abstract that one forgets how to conceptualize it shortly after finishing the task. This is an interesting and memorable way to put it, I like that.
- quantstats 5y agoThis article is relatively popular here (considering the topic). Two previous discussions about it: From 2015: https://news.ycombinator.com/item?id=9040266 https://news.ycombinator.com/item?id=9040266. From 2017: https://news.ycombinator.com/item?id=14405665 https://news.ycombinator.com/item?id=14405665. A more recent approach to visualizing high-dimensional data is the t-SNE algorithm, which I normally use together with PCA when exploring big data sets. If you're interested in the differences between both methods, here's a really good answer: https://stats.stackexchange.com/a/249520 https://stats.stackexchange.com/a/249520.
- rcar 5y agoPCA is a cool technique mathematically, but in my many years of building models, I've never seen it result in a more accurate model. I could see it potentially being useful in situations where you're forced to use a linear/logistic model since you're going to have to do a lot of feature preprocessing, but tree ensembles, NNs, etc. are all able to tease out pretty complicated relationships among features on their own. Considering that PCA also complicates things from a model interpretability point of view, it feels to me like a method whose time has largely passed.
- ivalm 5y agoIt is still a nice tool for projecting things (at least to visualize) where you expect the data to be on a lower dimensional hyperplane. I do agree in most cases t-SNE or UMAP are better (esp if you don’t care about distances).
- mcguire 5y agoBy definition, it's going to result in a less accurate model, unless you keep all of the dimensions or your data is very weird, right? And NNs are going to complicate your interpretability more?
- oivey 5y agoWhen/if used properly, no. The idea behind PCA is to find a set of features with far less dimensionality than the original data. The hope/intent with this sort of approach is that any more fitted features are just fitting noise.
- hervature 5y agoFor people who are curious, the GP is correct when it comes to fitting the training data. Recall, with enough parameters, we can get 100% on training. The parent’s comment is about testing/validation where we want to avoid overfitting so removing the least important parameters can be helpful.
- zmk_ 5y agoNot if many columns in your data are driven by some common latent factors.
- baron_harkonnen 5y ago> Considering that PCA also complicates things from a model interpretability point of view This is a strange comment since my primary usages of PCA/SVD is as a first step in understanding latent factors which are driving the data. Latent factors typically involve all of the important things that anyone running a business or deciding policy care about: customer engagement, patient well being, employee hapiness, etc all represent latent factors. If you have ever wanted to perform data analysis and gain some exciting insight into explaining user behavior, PCA/SVD will get you there pretty quickly. It is one of the most powerful tools in my arsenal when I'm working on a project that requires interoperability. The "loadings" in PC and the V matrix in SVD both contain information about how the original feature space correlates with the new projection. This can easily show thing things like "User's who do X,Y and NOT Z are more likely to purchase". Likewise in LSA (Latent Semantic Analysis/indexing) on a Term-Frequency matrix you will get a first pass at semantic embedding. You'll notice, for example, that "dog" and "cat" will project onto the new space in a common PC which can be used to interpret "pets". > I've never seen it result in a more accurate model. I could see it potentially being useful in situations where you're forced to use a linear/logistic model PCA/SVD are a linear transformation of the data and shouldn't give you any performance increase on a linear model. However they can be very helpful in transforming extremely high dimensional, sparse vectors into lower dimensional, dense representations. This can provide a lot of storage/performance benefits. > NNs, etc. are all able to tease out pretty complicated relationships among features on their own. PCA is literally identical to an autoencoder minimizing the MSE with no non-linear layers. It is a very good first step towards understanding what your NN will eventually do. After all, all NNs perform a non-linear matrix transformation so that your final vector space is ultimately linearly separable.
- rsj_hn 5y agoI put the four dots on the corners of a square and the fifth in the center. This results in the same square in the PCA pane but rotated about 45 degrees. Then, if you take one of the dots on the square corner and move it ever so sligthly in and out, you see the PCA square wildly rotating. Pretty cool to demonstrate sensitivity to small changes in the inputs.
- kwhitefoot 5y agoVery interesting. It would have been even better if there had been a link to an explanation of how PCA is performed.
- baron_harkonnen 5y agoIf you know a bit of linear algebra the transformation is surprisingly intuitive. Your goal is to create a set of orthogonal vectors, each that captures the highest amount of variance in the original data (the assumption is that variance is where the most information is). This is achieved by performing Eigen-decomposition on the Covariance matrix of the original data. Essentially you are learning the eigenvectors of the covariance matrix, ordered by their eigenvalues.
- bigdict 5y agoOr the singular vectors of the zero-centered data, ordered by singular values.
- onurcel 5y agoHere is the best PCA explanation I ever read on the web: https://stats.stackexchange.com/questions/2691/making-sense-of-principal-component-analysis-eigenvectors-eigenvalues/140579#140579 https://stats.stackexchange.com/questions/2691/making-sense-...
- saeranv 5y agoSeconded. This is exactly the same stackexchange post I thought of as well.
- knolan 5y agoI’ve always like this example from the Matlab documentation: https://www.mathworks.com/help/stats/biplot.html https://www.mathworks.com/help/stats/biplot.html The use of car data and how it correlates is a good way of showing how PCA treats each variable. For example acceleration and weight are negatively correlated while displacement and horsepower are highly correlated.
- melling 5y agoAnyone using Matlab these days? Seems a bit niche. I used Octave in Andrew Ng’s ML class and wanted to learn more Matlab so I signed up for a Coursera course on it. It’s a nice tool.
- deleted 5y ago[deleted]
- catillac 5y agoMatlab is great. I won’t hire anyone who only has experience in it though, like I might hire someone who only has python numerical analysis tools experience. It’s brittle and difficult to put into production in my experience.
- ingqondo 5y agoI'm sure it's still very popular in engineering. It's still very much the lingua franca for numerical methods applied to engineering problems - there are decades of materials solving engineering problems written in MATLAB.
- Sinidir 5y agoQuestion: is there any difference between the highest variance dimension pca finds and a line that linear regression would find?
- hsiang_jih_kueh 5y agoif recall yeah there probably will be. linear regression minimises the vertical distance of a point to the regression line whereas PCA minimises the orthogonal distance of the point to the line.
- osipov 5y agoLinear regression uses a measure of an "error" for every data point. Visually, the error is the vertical difference between a data point and the line/plane of linear regression. In contrast, PCA measures the distance from the data point along the line perpendicular to the PCA axis. The PCA distance is also known as a "projection". There is something known as orthogonal regression (total least squares) which uses the same measure as PCA. Unfortunately it doesn't work well across incompatible variables.
- atum47 5y agoCool animations. Statsquest has a really well made video about PCA on youtube, recommend.
- jschveibinz 5y agoEven simpler without the math: Let’s say you eat a piece of cake. You say, “hmm, salty, sweet, fruity, nice texture.” Let’s call those attributes the “principal components” of the cake from your point of view. Are there others? Maybe you could have also said “moist, or fluffy,” but you didn’t because those weren’t as obvious, so not as important. When the cake was made, according to the recipe, there were no instructions on how to add “sweet” or “fruity.” Instead, there was a list of ingredients: sugar, vanilla, lemon juice, flour, water, baking powder, etc. The mixture of these ingredients in the quantities dictated (plus the baking) resulted in the cake having the characteristics that you described. Some of the characteristics have a strong reliance on just one or two ingredients, e.g. “sweet” with “sugar,” and some characteristics are the result of subtle combinations of many ingredients, e.g. “texture.” The list of characteristics (principal components) definitely describe the cake, but in a more convenient and relevant way. You don’t need the whole list of ingredients to describe the cake. This is what makes principal component analysis useful.
- nxpnsv 5y agoIt's a nice start, but are sweet and fruity orthogonal components :)
- jschveibinz 5y agoOh you just had to go there :) Sure, let’s say that “sweet” and “fruity” are completely independent.
- usgroup 5y agoI think PCA is a good reason to learn enough linear algebra to understand PCA. It means learning about basis, rank, rank approximation, orthogonality, eigenvectors, spectral decomposition, etc. There's a whole iceberg of concepts that go into actually understand PCA without which PCA is not really understood.
- peteretep 5y agoToday I learned the toy recommendations algorithm I sometimes day dream about has a name...
- foota 5y agoI was thinking the other night, PCA can be on used on images for compression, what would it look like if you took two images, combined into pairs the principle components, and then lerped between them as a transition effect.
- jazzyjackson 5y agoLookup “data moshing“ and see if it’s what you’re thinking
- foota 5y agoNot really, I was thinking that by gradually shifting between the principal components of the image you could subtly shift from one to the other, but it might just look like visual garbage instead :) maybe like start with the lowest components and then gradually move to the strongest components.