3 ms·
Curious. How does freezing layers save you memory? Does it save compute time much? I understand the frozen layers do not need gradients to be stored?
by lostmsu 5y ago
Curious. How does freezing layers save you memory? Does it save compute time much?
I understand the frozen layers do not need gradients to be stored?
- minimaxir 5y agoEssentially yes. That technique is not exclusive to ruDALL-E; large models often freeze early layers and train lower layers only due to VRAM constraints.
- lostmsu 5y agoOh, right, only freezing early layers makes sense. I was thinking you froze inner ones, but gradients would need to be computed and kept for them to backpropagate to the unfrozen early ones.