3 ms·
A very curious effect of ruDALL-E is that the finetuning works on small datasets with unexpectedly good results. The Sneakers example they note in this article
by minimaxir 5y ago
A very curious effect of ruDALL-E is that the finetuning works on small datasets with unexpectedly good results. The Sneakers example they note in this article is on about ~10k images.
As an experiment, I finetuned ruDALL-E on about 1000 images of Pokemon and generated from that, which yielded incredible results that went viral: https://twitter.com/minimaxir/status/1470913487085785089 https://twitter.com/minimaxir/status/1470913487085785089
I then tried finetuning ruDALL-E on 1 Pokemon, yet still good/horrifying results: https://twitter.com/minimaxir/status/1474913997807755268 https://twitter.com/minimaxir/status/1474913997807755268
Unfortunately it's still a convoluted process to finetune ruDALL-E; I hope they end up releasing a smaller model to make it possible to do on a smaller/free GPU. (if they do, I'll release a streamined Colab notebook + blog post on how to do it)
- etaioinshrdlu 5y agoHow much GPU RAM and time does it currently take to fine-tune the current model?
- minimaxir 5y agoEssentially all of a 16GB GPU VRAM, even with some layers frozen. The more diverse the input images, the longer/more epochs the finetuning process should take in order to get stable results. The first Pokemon model was trained for about 4.5 hours; the one-shot model was about 2 minutes.
- lostmsu 5y agoCurious. How does freezing layers save you memory? Does it save compute time much? I understand the frozen layers do not need gradients to be stored?
- minimaxir 5y agoEssentially yes. That technique is not exclusive to ruDALL-E; large models often freeze early layers and train lower layers only due to VRAM constraints.
- lostmsu 5y agoOh, right, only freezing early layers makes sense. I was thinking you froze inner ones, but gradients would need to be computed and kept for them to backpropagate to the unfrozen early ones.
- deleted 5y ago[deleted]
- deleted 5y ago[deleted]
- smusamashah 5y agoDid you try feeding it with something else f.e. human faces only? It looks like it should already be able to generate faces as good as Nvidia StyleGAN. Pikachu variations from a single image are still way too good for anything we have seen before. It apparently understood the face part well and applied variations, for the rest it did as much as it understood. What else did you try after Pokémon and can you share those results?