2 ms·
I never tried finetuning Llama or Meta's newly released CodeLlama 70b on 1x H100 80GB card, but people have confirmed it works! On bsz=1, seqlen=16K, it just g
by danielhanchen 3y ago
I never tried finetuning Llama or Meta's newly released CodeLlama 70b on 1x H100 80GB card, but people have confirmed it works!
On bsz=1, seqlen=16K, it just gits on a 80GB card with 4bit QLoRA, since Unsloth reduces VRAM use by 70% and makes finetuning 2x faster! CodeLlama 70b / Llama 70b has 80 layers, a hd of 8192 and Swiglu of 28672. Vocab size is 32K.
1) 4bit weights of 70b takes around 40GB ish of VRAM.
2) Backprop saved tensors takes 80layers * 8192 * bsz(1) * 16K * 2bytes = 20.5GB VRAM.
3) Internal Attention gradients take 8192*5*1*16K*2bytes = 1.3GB VRAM.
4) Internal Swiglu gradients take 28672*3*1*16K*2bytes = 2.7GB VRAM.
5) Logits take 32K*1*16K*2bytes = 1GB VRAM.
6) LoRA weights rank 64 for attention QKVO take 64*8192*80layers*2bytes*4*2 = 640MB VRAM.
7) LoRA weights for MLP gate up down take 64*28672*80layers*2bytes*3*2 = 1.7GB VRAM.
8) LoRA gradients f32 take double so 2.3GB * 2 = 4.6GB VRAM.
9) Layernorm and other residuals internally take 3*8192*1*16K*2bytes = 1GB VRAM.
So all in all, 40 + 20.5 + 1.3 + 2.7 + 1 + 2.3 + 4.6 + 1 = 73.4GB of VRAM!!
Plus memory fragmentation, it can fit in under 80GB of VRAM!