3 ms·
Agreed that others should implement it as well but coercing llama to output results with matching grammar needs work.
by alhaad 2y ago
Agreed that others should implement it as well but coercing llama to output results with matching grammar needs work.
- _flux 2y agoWhat kind of work? I've only given it a short try before moving to Ollama that doesn't have it, but it seemed to have worked there. (With ollama I need to use a retry system.) edit: I researched a bit and apparently it can reduce performance, plus the streaming mode fails to report incorrect grammars. Overall these don't seem like deal-breakers.