2 ms·
> Do these memory management improvements make it possible to fine-tune a larger one? Unfortunately not yet; I need to implement gradient checkpointing first.
by minimaxir 6y ago
> Do these memory management improvements make it possible to fine-tune a larger one?
Unfortunately not yet; I need to implement gradient checkpointing first. Memory-wise, the results for finetuning 124M are promising (<8 GB VRAM when it used to take about 12 GB VRAM with gpt-2-simple)