4 ms·
most of the time when a local model feels dumb its not the quant, its the chat template. a lot of gguf mints just drop the template from the metadata and the ru
by anotherCodder 1mo ago
most of the time when a local model feels dumb its not the quant, its the chat template. a lot of gguf mints just drop the template from the metadata and the runtime silently falls back to chatml. model still talks fine so nobody notices, it just gets noticeably dumber. got burned by this myself serving qwen, now i grep the gguf for the template tokens before i blame anything else. second place is sampling, people run whatever defaults their ui ships instead of what the vendor recommends and then compare that to benchmark numbers that were run greedy or with the official settings
- washadjeffmad 1mo agoI've been comparing against TextGen and llama.cpp while I port to LocalAI and have been surprised by what's happening over the API, even with the defaults and jinja. It's been a fair reminder not to eschew familiarizing myself with the repos.
- dannyw 1mo agoNothing beats the classic of figuring out something yourself with your brain, but I also like dictating to LLMs a stream of consciousness with what I'm interested in (while forcing it to NOT give any answers or opinions), and getting back file names it suggests I look at and explore. Modern frontier LLMs can still be used as rubber ducks, and it's a great.