3 ms·

This is great and all. But LLM's are usually very good at following grammars. I rarely see LLM generating code that is OOD. Ofc, this is only true for popular

This is great and all. But LLM's are usually very good at following grammars. I rarely see LLM generating code that is OOD. Ofc, this is only true for popular language (JSON/Python/Java, etc), I can see how this is handy for more niche and in house DSL. You still need quite a lot of prompt engineering to get desired outputs, this just add another layer of output verification IMO. But does it really save much as comparing to get the output then parse and reject the output that doesn't follow the grammar? Might be debateable. But great work regardless.

mpalmer 3y ago
> this just add another layer of output verification IMO. It's not verifying the output after it's done, it's constraining the output as it's generated. > But does it really save much as comparing to get the output then parse and reject the output that doesn't follow the grammar? Might be debateable. I don't think it's debatable at all. Forcing the model to conform to a grammar during generation means there is never a need to discard and regenerate because it got the grammar wrong. Think of the compute involved in generating the whole output and then re-generating if it's non-conformant. There is no comparison.
- karmasimida 3y ago
  > It's not verifying the output after it's done, it's constraining the output as it's generated. It is verifying, by filtering only tokens that allowed by the grammar. I think we are talking about the same thing.