4 ms·

A key point that I feel is missing from this explanation is that the concept of distance becomes almost meaningless in very high dimensional space [1]. PCA is a

A key point that I feel is missing from this explanation is that the concept of distance becomes almost meaningless in very high dimensional space [1]. PCA is a means of dimensionality reduction, but the reason you want to reduce dimensions is so that you can measure and compare the distance between pairs of samples. [1] https://en.m.wikipedia.org/wiki/Clustering_high-dimensional_data https://en.m.wikipedia.org/wiki/Clustering_high-dimensional_...

yorwba 8y ago
The "meaninglessness" of distances in high-dimensional spaces, as defined in that Wikipedia article (i.e. the maximum and minimum distances of points becoming relatively close to each other) only happens when all those dimensions are independent. If the data is actually low-dimensional, such that PCA loses no information (trivial case: tack a billion zeros to all your vectors) then it won't change the distances at all, so it doesn't matter whether you compute them in the high-dimensional space or the low-dimensional projection obtained with PCA. PCA works under the assumption that you're dealing with a low-dimensional signal that was projected into a high-dimensional space and then corrupted by low-magnitude high-dimensional noise. By taking only the top k components, you filter out the noise and get a better signal. But if you have a low-magnitude high-dimensional signal corrupted by highly-correlated noise, then naively applying PCA will filter out the signal and leave you with only noise. So whether PCA makes your data more meaningful or less really depends on whether that assumption is satisfied or not. Dimensionality reduction is no silver bullet.
- jqgatsby 8y ago
  That's very interesting. Can you please clarify "if you have a low-magnitude high-dimensional signal corrupted by highly-correlated noise"? I don't understand what "highly-correlated noise" means. I thought noise was uncorrelated to the signal by definition. Also, I understand that PCA is not scale invariant, but I'm having trouble relating that to "low-magnitude, high-dimensional signal". Thank you!
  - yorwba 8y ago
    Noise can be correlated with itself. For example, if you're using a pressure sensor to detect whether the person carrying it has fallen, then the baseline atmospheric pressure is noise, while the signal is in relatively small changes corresponding to a meter or so of height difference. Because atmospheric pressure changes only slowly, it is highly correlated across successive measurements. If you did PCA on timelines recorded under different atmospheric pressure conditions, that would appear as the top component. I used that example because something similar actually happened (although there's no indication PCA was used): https://semiengineering.com/training-a-neural-network-to-fall/ https://semiengineering.com/training-a-neural-network-to-fal...
    - mikorym 8y ago
      This is, I think, another example where one may consider to discard factor 1 and to look only at factor 2 and 3. (Of course, a better approach if possible would be baseline normalisation.)
  - abakker 8y ago
    I think of survey responses as being low-magnitude, high signal sometimes. Take a list of 100 questions all answered between 1-5 (disagree -agree). Sometimes, PCA can help you group these statements into like concepts. Which means instead of having to look at variation across all 100 statements you only need to look at a handful. However, in doing so, you might miss out on an interesting pattern in a subset of the data where a given subset of respondents are consistently answering a single choice differently. Maybe in aggregate that signal doesn’t show up, and so you would wash it out in PCA. Obviously, in this case, each of the 100 questions is a dimension.
mikorym 8y ago
This distance that you speak of falls under my use of the word "classify" which now in retrospect is a poor choice of word. However, as explained by another comment, it does not always become "meaningless". In fact, it becomes variable. For example, if you are interested in secondary structure in your data, you can plot factors 2, 3, 4 or higher against another factor and often the change in pairwise distance at different factors is meaningful.
- mikorym 8y ago
  For example, I've had an example where factor 1 would classify low resolution parameters and factor 2 and further are the only factors of relevance. When one examined the data, some parameters were in 40 square kilometer boxes and others in more like 0.5 square kilometer boxes. Factor 1 would tell you which parameters are in the 40 square kilometer boxes. Another interesting example is that one can use this to for example counteract the effects of inflation on sales prices.

yorwba 8y ago

The "meaninglessness" of distances in high-dimensional spaces, as defined in that Wikipedia article (i.e. the maximum and minimum distances of points becoming relatively close to each other) only happens when all those dimensions are independent. If the data is actually low-dimensional, such that PCA loses no information (trivial case: tack a billion zeros to all your vectors) then it won't change the distances at all, so it doesn't matter whether you compute them in the high-dimensional space or the low-dimensional projection obtained with PCA. PCA works under the assumption that you're dealing with a low-dimensional signal that was projected into a high-dimensional space and then corrupted by low-magnitude high-dimensional noise. By taking only the top k components, you filter out the noise and get a better signal. But if you have a low-magnitude high-dimensional signal corrupted by highly-correlated noise, then naively applying PCA will filter out the signal and leave you with only noise. So whether PCA makes your data more meaningful or less really depends on whether that assumption is satisfied or not. Dimensionality reduction is no silver bullet.

jqgatsby 8y ago
That's very interesting. Can you please clarify "if you have a low-magnitude high-dimensional signal corrupted by highly-correlated noise"? I don't understand what "highly-correlated noise" means. I thought noise was uncorrelated to the signal by definition. Also, I understand that PCA is not scale invariant, but I'm having trouble relating that to "low-magnitude, high-dimensional signal". Thank you!
- yorwba 8y ago
  Noise can be correlated with itself. For example, if you're using a pressure sensor to detect whether the person carrying it has fallen, then the baseline atmospheric pressure is noise, while the signal is in relatively small changes corresponding to a meter or so of height difference. Because atmospheric pressure changes only slowly, it is highly correlated across successive measurements. If you did PCA on timelines recorded under different atmospheric pressure conditions, that would appear as the top component. I used that example because something similar actually happened (although there's no indication PCA was used): https://semiengineering.com/training-a-neural-network-to-fall/ https://semiengineering.com/training-a-neural-network-to-fal...
  - mikorym 8y ago
    This is, I think, another example where one may consider to discard factor 1 and to look only at factor 2 and 3. (Of course, a better approach if possible would be baseline normalisation.)
- abakker 8y ago
  I think of survey responses as being low-magnitude, high signal sometimes. Take a list of 100 questions all answered between 1-5 (disagree -agree). Sometimes, PCA can help you group these statements into like concepts. Which means instead of having to look at variation across all 100 statements you only need to look at a handful. However, in doing so, you might miss out on an interesting pattern in a subset of the data where a given subset of respondents are consistently answering a single choice differently. Maybe in aggregate that signal doesn’t show up, and so you would wash it out in PCA. Obviously, in this case, each of the 100 questions is a dimension.

mikorym 8y ago

This distance that you speak of falls under my use of the word "classify" which now in retrospect is a poor choice of word. However, as explained by another comment, it does not always become "meaningless". In fact, it becomes variable. For example, if you are interested in secondary structure in your data, you can plot factors 2, 3, 4 or higher against another factor and often the change in pairwise distance at different factors is meaningful.

mikorym 8y ago
For example, I've had an example where factor 1 would classify low resolution parameters and factor 2 and further are the only factors of relevance. When one examined the data, some parameters were in 40 square kilometer boxes and others in more like 0.5 square kilometer boxes. Factor 1 would tell you which parameters are in the 40 square kilometer boxes. Another interesting example is that one can use this to for example counteract the effects of inflation on sales prices.