4 ms·
How much GPU RAM and time does it currently take to fine-tune the current model?
by etaioinshrdlu 5y ago
How much GPU RAM and time does it currently take to fine-tune the current model?
- minimaxir 5y agoEssentially all of a 16GB GPU VRAM, even with some layers frozen. The more diverse the input images, the longer/more epochs the finetuning process should take in order to get stable results. The first Pokemon model was trained for about 4.5 hours; the one-shot model was about 2 minutes.
- lostmsu 5y agoCurious. How does freezing layers save you memory? Does it save compute time much? I understand the frozen layers do not need gradients to be stored?
- minimaxir 5y agoEssentially yes. That technique is not exclusive to ruDALL-E; large models often freeze early layers and train lower layers only due to VRAM constraints.
- lostmsu 5y agoOh, right, only freezing early layers makes sense. I was thinking you froze inner ones, but gradients would need to be computed and kept for them to backpropagate to the unfrozen early ones.