2 ms·
On my Mac M1 Max 32GB of ram, Vicuna 13b (GGUF model) at 4bit consumes around 8GB of ram in Oobabooga. Tried turning on mlock and upping thread count to 6, but
by vunderba 3y ago
On my Mac M1 Max 32GB of ram, Vicuna 13b (GGUF model) at 4bit consumes around 8GB of ram in Oobabooga.
Tried turning on mlock and upping thread count to 6, but it's still rather slow at around 3 tokens / sec.