3 ms·
> A 7B model @ Q4 needs roughly 6.5GB of memory. Mistral 7B q4 seems to use ~ 5Gb VRAM A few months ago I could not run Llama 7B q4 on my laptop's rtx 3070 (8
by tarruda 3y ago
> A 7B model @ Q4 needs roughly 6.5GB of memory.
Mistral 7B q4 seems to use ~ 5Gb VRAM
A few months ago I could not run Llama 7B q4 on my laptop's rtx 3070 (8gb VRAM), but Mistral 7B it runs very well and still has approx 2gb VRAM left even with a big context.
Not sure if this is an optimization on inference algorithms or if Mistral 7B is just more efficient