3 ms·
llama.cpp has the feature to enforce a certain structure from the LLM (but you still need to have the LLM be able to produce that structure and it's beneficial
by _flux 3y ago
llama.cpp has the feature to enforce a certain structure from the LLM (but you still need to have the LLM be able to produce that structure and it's beneficial to prompt it towards the exact result). So this particular story doesn't need to be the same among all LLMs.
https://github.com/ggerganov/llama.cpp/blob/master/grammars/README.md https://github.com/ggerganov/llama.cpp/blob/master/grammars/...
- simonw 3y agollama.cpp grammars will get you results that definitely conform to your grammar, but that doesn't guarantee that they'll be semantically correct - the model could still hallucinate details incorrectly while returning valid JSON.
- _flux 3y agoIndeed it's not the solution to LLM hallucinations—as far as I know nobody knows a solution to it. But it is the solution to needing to run the model again and checking the format of the output to ensure that it conforms to you expectations.