3 ms·
It looks like it's pretty resistant to quantization. ollama 4bit 7B doesn't work very well, but the 16bit 2B does
by brrrrrm 3y ago
It looks like it's pretty resistant to quantization. ollama 4bit 7B doesn't work very well, but the 16bit 2B does
- petercooper 3y agoThat's useful to know. My experiments with the 4bit 7B currently tagged for use on ollama are not going well at all. Lots of refusals and junk. Downloading 7b-instruct-fp16 now! :-) (Update: Yes, much better, though much slower too, of course.)
- janwas 3y agoFYI we have an experimental 4.5-bit format (NUQ) in gemma.cpp, our C++ inference engine, which from my testing works reasonably well.
- petercooper 3y agoThanks, I'm going to give that a go too. The good news is the very latest Ollama update (0.1.27) has resolved things (though I also had to redownload the quant model for 7B).