4 ms·

OK, I am not well-versed in machine learning, so I need to ask: How is it really different, in an essential way, from using Eigenfaces[1] for the same task? T

OK, I am not well-versed in machine learning, so I need to ask: How is it really different, in an essential way, from using Eigenfaces[1] for the same task? There's a paper from year 2000 that does exactly that[2]; you can see the results on page 5. Now, the results in the linked article are way more impressive than the Eigenfaces-generated random faces, but from skimming the text, it seems like the principles are the same: dimensionality reduction from a high-dimensional space into a space where the axis are "meaningful" components (in Eigenfaces, the reduction is done using SVD). Eigenfaces is a linear approach; there's also a multilinear version, TensorFaces by Vasilescu et al[3][4]. I wonder if similar quality images can be obtained using this approach. [1]https://en.wikipedia.org/wiki/Eigenface https://en.wikipedia.org/wiki/Eigenface [2]https://link.springer.com/content/pdf/10.3758/BF03207802.pdf https://link.springer.com/content/pdf/10.3758/BF03207802.pdf [3]http://alumni.media.mit.edu/~maov/mica/mica05.pdf http://alumni.media.mit.edu/~maov/mica/mica05.pdf [4]http://web.cs.ucla.edu/~dt/papers/eccv02/eccv02.pdf http://web.cs.ucla.edu/~dt/papers/eccv02/eccv02.pdf [5]https://en.wikipedia.org/wiki/Convex_set https://en.wikipedia.org/wiki/Convex_set

skinner_ 8y ago
In both cases, there's a low-dimensional latent space, and there's a pixel space of images. In the case of Eigenfaces, the correspondence between the two is linear. That's not a bad approximation, but "make him smile" or "make him bold" is obviously a very nonlinear operation. Eigenfaces approximates it with something linear, and that's just not very good. The justification for all the added nonlinearity in the mapping is that this operation really becomes linear in the latent space.
mlboss 8y ago
As you said the result is way more impressive. That is the main theme when you try to compare deep learning based approaches to traditional ML approaches especially in unstructured data(images, speech, text). Also, deep learning based approaches don't require too much fiddling with features. Main work involves coming up with NN architecture and loss function.
quadcore 8y ago
Deep learning will do eigenfaces of eigenfaces of... it's eigenfaces all the way down. Actually it's not all the way down, there is only a few hidden layers.
halflings 8y ago
Compare the images used for eigenfaces to those that GANs synthesize: different angles, facial expressions, glasses vs no glasses, etc.
currymj 8y ago
You are right that they are very closely related. For variational autoencoders (one baseline technique for this sort of thing) if you make your neural network have one layer without a nonlinearity, and train it, it ends up minimizing the same objective as PCA (i.e. finding the eigenfaces). I believe this is also true of GANs where you similarly restrict the generator and discriminator to be very simple. I bet there is a nonlinear non-NN approach that could perform well, but we may not have the investment in hardware, well-optimized algorithms, etc to train big models fast. edit: here's a paper that connects GAN to PCA in a simple case, among many other things. not the easiest to follow, though. https://arxiv.org/pdf/1710.10793.pdf https://arxiv.org/pdf/1710.10793.pdf

skinner_ 8y ago

In both cases, there's a low-dimensional latent space, and there's a pixel space of images. In the case of Eigenfaces, the correspondence between the two is linear. That's not a bad approximation, but "make him smile" or "make him bold" is obviously a very nonlinear operation. Eigenfaces approximates it with something linear, and that's just not very good. The justification for all the added nonlinearity in the mapping is that this operation really becomes linear in the latent space.

mlboss 8y ago

As you said the result is way more impressive. That is the main theme when you try to compare deep learning based approaches to traditional ML approaches especially in unstructured data(images, speech, text). Also, deep learning based approaches don't require too much fiddling with features. Main work involves coming up with NN architecture and loss function.

quadcore 8y ago

Deep learning will do eigenfaces of eigenfaces of... it's eigenfaces all the way down. Actually it's not all the way down, there is only a few hidden layers.

halflings 8y ago

Compare the images used for eigenfaces to those that GANs synthesize: different angles, facial expressions, glasses vs no glasses, etc.

currymj 8y ago

You are right that they are very closely related. For variational autoencoders (one baseline technique for this sort of thing) if you make your neural network have one layer without a nonlinearity, and train it, it ends up minimizing the same objective as PCA (i.e. finding the eigenfaces). I believe this is also true of GANs where you similarly restrict the generator and discriminator to be very simple. I bet there is a nonlinear non-NN approach that could perform well, but we may not have the investment in hardware, well-optimized algorithms, etc to train big models fast. edit: here's a paper that connects GAN to PCA in a simple case, among many other things. not the easiest to follow, though. https://arxiv.org/pdf/1710.10793.pdf https://arxiv.org/pdf/1710.10793.pdf