23 ms·
If anyone needs a more powerful constrain outputs, llama.cpp support gbnf: https://github.com/ggerganov/llama.cpp/blob/master/grammars/README.md https://github
by rdescartes 2y ago
If anyone needs a more powerful constrain outputs, llama.cpp support gbnf:
https://github.com/ggerganov/llama.cpp/blob/master/grammars/README.md https://github.com/ggerganov/llama.cpp/blob/master/grammars/...
- jimmySixDOF 2y agoThats is exactly what they are using
- sa-code 2y agoThis is amazing, thank you for the link
- lolinder 2y agoHave you found the output for arbitrary grammars to be satisfactory? My naive assumption has been that these models will produce better JSON than other formats simply by virtue of having seen so much of it.
- rdescartes 2y agoIf you want to get a good result, the grammar should be following the expect output from the prompt, especially if you use a small model. Normally I would manually fine-tune the prompt to output the grammar format first, and then apply the grammar in production.
- throwaway314155 2y agoWho would downvote this perfectly reasonable question? edit: Nm
- dcreater 2y agoHow is it more powerful?
- evilduck 2y agoGrammars don't have to just be JSON, which means you could have it format responses as anything with a formal grammar. XML, HTTP responses, SQL, algebraic notation of math, etc.
- deleted 2y ago[deleted]