3 ms·
People that run 7B 4bit is probably because they don't have the memory for more. And there are tests showing that quantizing a bigger model is always better tha
by redox99 3y ago
People that run 7B 4bit is probably because they don't have the memory for more. And there are tests showing that quantizing a bigger model is always better than running a smaller model full precision (i.e LlaMa 13B 4bit > 7B fp16)