3 ms·
FYI we have an experimental 4.5-bit format (NUQ) in gemma.cpp, our C++ inference engine, which from my testing works reasonably well.
by janwas 3y ago
FYI we have an experimental 4.5-bit format (NUQ) in gemma.cpp, our C++ inference engine, which from my testing works reasonably well.
- petercooper 3y agoThanks, I'm going to give that a go too. The good news is the very latest Ollama update (0.1.27) has resolved things (though I also had to redownload the quant model for 7B).