4 ms·
8GB of memory is not currently suitable for LLM use. A 7B model @ Q4 needs roughly 6.5GB of memory. I have a 64GB M1. If my memory pressure is high and I star
by SparkyMcUnicorn 3y ago
8GB of memory is not currently suitable for LLM use.
A 7B model @ Q4 needs roughly 6.5GB of memory.
I have a 64GB M1. If my memory pressure is high and I start up a model that goes into swap, the machine becomes unbearably slow.
- brucethemoose2 3y agoMore than that for the 8K+ context people are running now.
- tarruda 3y ago> A 7B model @ Q4 needs roughly 6.5GB of memory. Mistral 7B q4 seems to use ~ 5Gb VRAM A few months ago I could not run Llama 7B q4 on my laptop's rtx 3070 (8gb VRAM), but Mistral 7B it runs very well and still has approx 2gb VRAM left even with a big context. Not sure if this is an optimization on inference algorithms or if Mistral 7B is just more efficient