3 ms·
> If you change the units that one of the variables is measured in, it will change all the "principal components" By the way that's worded, it sounds like a ca
by kyzyl 14y ago
> If you change the units that one of the variables is measured in, it will change all the "principal components"
By the way that's worded, it sounds like a case of sensitivity rather than the method. If you change one of the variables to a unit that is way out of scale, then it's quite possible that the results of PCA, and many other methods, will change. But that's because it's not scale invariant, and so if you want good results you need to present your variables in the same units, and/or in some normalized format (zscored, etc.) where the scale of one unit doesn't blow the others out of the water.
These methods are not magical, and they are not intelligent. They do not know what they're looking at, so it's your job to feed them something reasonable.
That said, if you take some physically grounded data and change units, you won't recover different eigenvectors as long as your input data is good. The physics doesn't care about the units, or the coordinate system, or whether you use python or anything else.
- tgflynn 14y agoif you take some physically grounded data and change units, you won't recover different eigenvectors as long as your input data is good. I don't think that's accurate. Consider a set of points distributed along a line in 2D. If you do PCA on these points you will find that the 1st eigenvector points along the line and the second is orthogonal to it. If you now rescale the axes so that x is measured in meters and y in light years the slope of the line will change and so will the the 2 PCA eigenvectors. However the relationship between the 2 eigenvectors and the distribution of the data points will remain the same. The first eigenvector will still point along the line and the second will still be orthogonal to it. In machine learning one is interested in the distribution of the data not in whatever units they happen to be measured in, hence I don't understand MacKay's objection.
- photon137 14y agoBut the principal eigenvectors can change if scales are changed - and the directions in which they may then point can be completely different.
- jessriedel 14y agoThe mathematical way to state this is that a choice of units is equivalent to a choice of an inner product in linear algebra( or, more generally, the choice of a metric tensor in differential geometry). Basically, the choice of an inner product defines what it means for something to be orthogonal in a space. Principal component analysis is predicated on a choice of inner product, since component directions are always chosen to be orthogonal. It's not clear what orthogonality could mean in a 2D plane where one direction is measured in inches and the other in tons, so naive PCA isn't appropriate in such a case. Others have mentioned in this thread that "whitening" the data before PCA fixes this problem, by removing cross-correlations. Presumably, in that case, the notion of orthogonality is taken from the statistical properties of the data. (Maybe it normalizes physical units like inches to the standard deviation of the data's distribution in inches?)