4 ms·

PCA typically implies that the high dimensional data (images) lie on a low dimensional linear manifold (the principal components). In this case the data of inte

PCA typically implies that the high dimensional data (images) lie on a low dimensional linear manifold (the principal components). In this case the data of interest (images of people in dresses) likely lies in a very non-linear manifold, so non-linear methods would likely result in much more appropriate models. This is part of why deep learning and tensor methods work so well on image classification tasks. Nonlinear methods would require much more data, unfortunately.

delhanty 11y ago
> This is part of why deep learning and tensor methods work so well on image classification tasks. That statement sounds informative. Do you have a good non-specialist's reference?
- jamessb 11y ago
  You might be interested by colah's (Christopher Olah's) blog: https://colah.github.io/ https://colah.github.io/ Two articles in particular are good introductions to looking at neural networks in terms of higher dimensional data lying on lower-dimesnional manifolds: Neural Networks, Manifolds, and Topology: https://colah.github.io/posts/2014-03-NN-Manifolds-Topology/ https://colah.github.io/posts/2014-03-NN-Manifolds-Topology/ Visualizing MNIST: An Exploration of Dimensionality Reduction: https://colah.github.io/posts/2014-10-Visualizing-MNIST/ https://colah.github.io/posts/2014-10-Visualizing-MNIST/
  - delhanty 11y ago
    Thank you for the useful links.
GFK_of_xmaspast 11y ago
I'd be interested to know how the singular values are distributed and how many you need before you can cut off the rest. The reconstruction example she shows looks pretty ok after only 70 components (out of presumably a lot lot more). Actually, I'd be even more interested in an NMF decomposition of the data, the weights for the 10-component approximation are [-17541.81, -12749.33, -3766.29, 2005.28, 4193.08, 6832.55, -6704.90, -2135.51, 1112.27, 7627.80]. and I wonder if a purely-additive approach will work better.
dthal 11y ago
Nonlinear methods can result in a smaller (lower-dimensional) representation, but they are non-linear so they are harder to use and usually require more data. On the other hand, PCA is easy and having 50 or more principal components is often not a problem, unless you are doing visualization. With the additional representational capacity from just keeping extra dimensions you can still get good reconstructions.
noelwelsh 11y ago
If your goal is to accurately recreate dresses, then yes, a non-linear method would work better. But, I don't think that makes this work uninteresting. For a start, PCA is very simple to implement and train, which gives it hackability that non-linear methods typically don't have. PCA makes a better tool if you just want to play around. Exploring non-realistic output is also interesting for its own sake. I find the "ghost" effect that PCA generates quite nice.

delhanty 11y ago

> This is part of why deep learning and tensor methods work so well on image classification tasks. That statement sounds informative. Do you have a good non-specialist's reference?

jamessb 11y ago
You might be interested by colah's (Christopher Olah's) blog: https://colah.github.io/ https://colah.github.io/ Two articles in particular are good introductions to looking at neural networks in terms of higher dimensional data lying on lower-dimesnional manifolds: Neural Networks, Manifolds, and Topology: https://colah.github.io/posts/2014-03-NN-Manifolds-Topology/ https://colah.github.io/posts/2014-03-NN-Manifolds-Topology/ Visualizing MNIST: An Exploration of Dimensionality Reduction: https://colah.github.io/posts/2014-10-Visualizing-MNIST/ https://colah.github.io/posts/2014-10-Visualizing-MNIST/
- delhanty 11y ago
  Thank you for the useful links.

GFK_of_xmaspast 11y ago

I'd be interested to know how the singular values are distributed and how many you need before you can cut off the rest. The reconstruction example she shows looks pretty ok after only 70 components (out of presumably a lot lot more). Actually, I'd be even more interested in an NMF decomposition of the data, the weights for the 10-component approximation are [-17541.81, -12749.33, -3766.29, 2005.28, 4193.08, 6832.55, -6704.90, -2135.51, 1112.27, 7627.80]. and I wonder if a purely-additive approach will work better.

dthal 11y ago

Nonlinear methods can result in a smaller (lower-dimensional) representation, but they are non-linear so they are harder to use and usually require more data. On the other hand, PCA is easy and having 50 or more principal components is often not a problem, unless you are doing visualization. With the additional representational capacity from just keeping extra dimensions you can still get good reconstructions.

noelwelsh 11y ago

If your goal is to accurately recreate dresses, then yes, a non-linear method would work better. But, I don't think that makes this work uninteresting. For a start, PCA is very simple to implement and train, which gives it hackability that non-linear methods typically don't have. PCA makes a better tool if you just want to play around. Exploring non-realistic output is also interesting for its own sake. I find the "ghost" effect that PCA generates quite nice.