7 ms·
Their killer feature is the --grammar option which restricts the logits the LLM outputs which makes them great for bash scripts that do all manner of NLP classi
by throwup238 3y ago
Their killer feature is the --grammar option which restricts the logits the LLM outputs which makes them great for bash scripts that do all manner of NLP classification work.
Otherwise I use ollama when I need a local LLM, vllm when I'm renting GPU servers, or OpenAI API when I just want the best model.
- verdverm 3y agoInteresting, you inspired me to https://github.com/jmorganca/ollama/issues/1507 https://github.com/jmorganca/ollama/issues/1507 I've also had good success instructing arbitrary grammars in the system prompt, though it doesn't work at the logits level, which can be helpful
- throwup238 3y agoThere's an old PR for it: https://github.com/jmorganca/ollama/pull/565 https://github.com/jmorganca/ollama/pull/565 (it just uses the underlying llama.cpp grammar feature which is what llamafile does)
- verdverm 3y agothanks for the pointer, I'll update my issue to reference that PR