3 ms·
can it be quantised further so it can run locally on a normal laptop of a developer?
by suyash 3y ago
can it be quantised further so it can run locally on a normal laptop of a developer?
- syntaxing 3y ago“Normal laptop” is kind of hard to gauge but if you have a M series MacBook with 16GB+ RAM, you will be able to run 7B comfortably and 13B but stretching your RAM (cause of the unified RAM) at 4 bit quantization. These go all the way down to 2 bit but I personally I find the model noticeably deteriorate anything below 4 bit. You can see how much (V)RAM you need here [1]. [1] https://github.com/ggerganov/llama.cpp#quantization https://github.com/ggerganov/llama.cpp#quantization
- totallywrong 3y agoJust started playing with this, there's a tool called ollama that runs Llama2 13B on my 16GB M1 Pro really smoothly with zero config.