4 ms·
Looks like you need about 120 GB to fine-tune the 65B model with this code at a sequence length of 512. How does the memory usage scale as the sequence length g
by Taek 4y ago
Looks like you need about 120 GB to fine-tune the 65B model with this code at a sequence length of 512. How does the memory usage scale as the sequence length grows?
- lxe 4y agoLots of VRAM, but the method is the same. Here's someone who finetuned llama-30b on alpaca dataset for example: https://github.com/deep-diver/Alpaca-LoRA-Serve https://github.com/deep-diver/Alpaca-LoRA-Serve
- lxe 4y ago> How does the memory usage scale as the sequence length grows? That's a good question. I was under the assumption it's linearly proportional, but I can test it out I guess.
- Taek 4y agoI suspect its linear with a small constant factor.
- arpowers 4y agoReally takes that much vram??
- ioedward 4y agoNormally people split up the model across multiple GPUs, i.e. model/tensor parallelism.
- albertzeyer 4y agoThe attention implementation is in a way that the memory scales quadratically with sequence lengths. Overall, this is still a small factor compared to just the model weights, but at some seq lengths, this would dominate. By using flash attention, you can get memory requirement down to scale linearly with sequence lengths.
- joshxyz 4y agoman i hope there are online calculators that lets people visualize the costs of training these things.