4 ms·
It needs more VRAM than just the model weights. With 6GB of VRAM, I got 44/65 layers loaded into VRAM. Has anyone tested 8GB yet?
by Dwedit 10d ago
It needs more VRAM than just the model weights. With 6GB of VRAM, I got 44/65 layers loaded into VRAM. Has anyone tested 8GB yet?
- kennywinker 10d ago“Without context” was me gesturing at that. Interesting you can’t fit the whole model tho - why is beyond my current understanding :)
- Dwedit 10d agoEven with a small context size (4096 at 64KB per token), that's like 256MB for the context. It's more than just the context that's eating up VRAM.