3 ms·
That's useful to know. My experiments with the 4bit 7B currently tagged for use on ollama are not going well at all. Lots of refusals and junk. Downloading 7b-i
by petercooper 3y ago
That's useful to know. My experiments with the 4bit 7B currently tagged for use on ollama are not going well at all. Lots of refusals and junk. Downloading 7b-instruct-fp16 now! :-) (Update: Yes, much better, though much slower too, of course.)
- janwas 3y agoFYI we have an experimental 4.5-bit format (NUQ) in gemma.cpp, our C++ inference engine, which from my testing works reasonably well.
- petercooper 3y agoThanks, I'm going to give that a go too. The good news is the very latest Ollama update (0.1.27) has resolved things (though I also had to redownload the quant model for 7B).