5 ms·
The central technique used in DeepFakes is fascinating. Initially I assumed they were using Generative Adversarial Networks (GANs). But — and someone can correc
by gdubs 7y ago
The central technique used in DeepFakes is fascinating. Initially I assumed they were using Generative Adversarial Networks (GANs). But — and someone can correct if I’m wrong — Deep Fakes use two Autoencoders.
So essentially you’re training a network how to compress an image down to a very tiny representation, and then uncompress it as accurately to the original as possible.
You train two of these: one for the original face, and one for the target face. Then, you compress with the “source” autoencoder, and then uncompress with the “target” autoencoder. And, voila, ‘source‘ face becomes ‘target’ face.
- Der_Einzige 7y agoAutoencoders aren't even the best dimensionality reduction technique. Where are UMAP powered "manifold" fakes?
- LolWolf 7y agoUMAP is, sadly, extraordinarily slow, plus the inverse mapping is not clear.
- Der_Einzige 7y agoThere are GPU implementations...
- LolWolf 7y agoStill doesn't fix the inverse mapping problem, which is done immediately by Autoencoders, by definition.
- Forge36 7y agoThat is a fascinating way to describe this.
- nojvek 7y agoMakes sense. You want to boil down the face to some sort of abstract representation of movement of lips, face alignment etc. Then put in different parameters and have it be photorealistic on the other side. Could do wonders for video chat.
- tralarpa 7y agoHow do you guarantee that the two trained autoencoders speak the "same language", i.e. the code learned by the autoencoder of the original face gives a reasonable output when uncompressed by the target autoencoder?
- claytonius 7y agoSomething like a CycleGAN and/ or weight sharing would do the trick
- orange3xchicken 7y agoThis is a good question. I'm not sure which paper the parent comment is referencing - and in general, DeepFakes is a very general term for a variety of techniques including GANs. The problem you describe is called the alignment problem and it's studied in several areas of applied ml: translation, multi-modal learning, etc. For example, the canonical way to train deep machine translation systems is to make use of large parallel corpora - large set of translation pairs. In the case where this data may not exist, for structurally similar languages, one approach is to learn unsupervised word embeddings independently (i.e. using word2vec) which only require monolingual data and to find a transformation that aligns the two spaces. In many cases, neural networks still struggle with this task. However, some work in the last few years has demonstrated that a few seed word-translations is all you need to apply simple algorithms that can learn a linear or simple nonlinear transformation that work pretty well! [1] Here is a method that proposes locally linear transformation maps for alignment: http://nakashole.com/papers/2018-emnlp-norm.pdf http://nakashole.com/papers/2018-emnlp-norm.pdf
- claytonius 7y agoI think you're right that most DeepFakes use Variational AutoEncoders (As described by the article) where the latent space is jointly distributed between the two domains, allowing for translation. For additional context: AutoEncoders are also frequently adversarially regularized so you're not wrong about GANs being involved. My understanding is that the most accurate implementations also have some manual feature engineering for more tractable problems like facial orientation. One of my favorite related papers: https://arxiv.org/pdf/1703.00848.pdf https://arxiv.org/pdf/1703.00848.pdf
- skizm 7y agoThe article describes this process in detail.