4 ms·
I thought they got like a logit mask that removes illegal next tokens, at least in llama.cpp and GBNF.
by tingletech 3mo ago
I thought they got like a logit mask that removes illegal next tokens, at least in llama.cpp and GBNF.
- ptx 3mo agoYou're probably right about that. It looks like llama.cpp lets you specify a grammar file or a JSON schema - is that what you mean? If the syntax of your DSL can be expressed that way, I suppose you wouldn't have to validate it externally and retry.
- tingletech 3mo agothe grammar support/json-schema applies to a whole response, but I was under the impression a similar logit guide rail can be applied when a tool call is opened.