3 ms·
llama.cpp does support something like this -- you can give it a grammar which restricts the set of available next tokens that are sampled over so in theory you
by dilap 2y ago
llama.cpp does support something like this -- you can give it a grammar which restricts the set of available next tokens that are sampled over
so in theory you could notice "```python" or whatever and then start restricting to valid python code. (in least in theory, not sure how feasible/possible it would be in practice w/ their grammar format.)
for code i'm not sure how useful it would be since likely any model that is giving you working code wouldn't be struggling w/ syntax errors anyway?
but i have had success experimentally using the feature to drive fiction content for a game from a smaller llm to be in a very specific format.
- notsylver 2y agoyeah, ive used llama.cpp grammars before, which is why i was thinking about it. i just think it'd be cool for llamafile to do basically that, but with included defaults so you could eg, require JSON output. it could be cool for prototyping or something. but i dont think that would be too useful anyway, most of the time i think you would want to restrict it to a specific schema, so i can only see it being useful for something like a tiny local LLM for code completion, but that would just encourage valid-looking but incorrect code. i think i just like the idea of restricting LLM output, it has a lot of interesting use cases
- dilap 2y agogotchya. i do think that is a cool idea actually -- LLMs tiny enough to do useful things with formally structured output but not big enough to nail the structure ~100% is probably not an empty set.