4 ms·
This is great, my employer just gave me a M1 laptop with only 16gb ram and I had to downgrade my 7B parameter local LLM’s to 3 bit quantizing, they’ve been surp
by yieldcrv 3y ago
This is great, my employer just gave me a M1 laptop with only 16gb ram and I had to downgrade my 7B parameter local LLM’s to 3 bit quantizing, they’ve been surprisingly okay!
In my personal machine at 64gb ram, I usually use 8x7B at Q5 or 70B at Q4
Its Mistral all the way down! Imagining Q1.58 that’s doing well makes me happy
- turnsout 3y agoQuantized 7B LLMs should work fine on your machine, though maybe you’re talking about speed?
- yieldcrv 3y ago7B works fine
- woadwarrior01 3y agoYou can run 4 bit quantized versions of SOLAR-10.7B and Llama 2 13B based models quite well on 16GB M1 laptops.
- FergusArgyll 3y agoYou shouldn't have to quantize it that much, maybe you're running a lot of other programs while running inference? Also, try using pure llama.cpp, AFAIK it's the least possible overhead
- regularfry 3y agoGetting more value out of phi-2-sized models is where you really want to be on lower-end M1's.