3 ms·
They're using the same trick OpenAI have been using for a while: they compile a grammar and then have that running as part of token inference, such that only to
by simonw 1y ago
They're using the same trick OpenAI have been using for a while: they compile a grammar and then have that running as part of token inference, such that only tokens that fit the grammar are selected as the next-token.
This trick has also been in llama.cpp for a couple of years: https://til.simonwillison.net/llms/llama-cpp-python-grammars https://til.simonwillison.net/llms/llama-cpp-python-grammars
- huevosabio 1y agoYea, and now there are mature OSS solutions with outlines and xgrammar, so it makes even more weird that only now do we have this supported by Anthropic.
- minimaxir 1y agoMore info on Claude's grammar compiling: https://docs.claude.com/en/docs/build-with-claude/structured-outputs#grammar-compilation-and-caching https://docs.claude.com/en/docs/build-with-claude/structured...
- causal 1y agoI reaaaaally wish we could provide an EBNF grammar like llama.cpp. JSON Schema has much fewer use cases for me.
- jawiggins 1y agoHow sure are you that OpenAI is using that? I would have suspected it too, but I’ve been struggling with OpenAI returning syntactically invalid JSON when provided with a simple pydantic class (a list of strings), which shouldn’t be possible unless they have a glaring error in their grammar.
- gradys 1y agoYou might be using JSON mode, which doesn’t guarantee a schema will be followed, or structured outputs not in strict mode. It is possible to get the property that the response is either a valid instance of the schema or an error (eg for refusal)
- jawiggins 1y agoHow do you activate strict mode when using pydantic schemas? It doesn't look like that is a valid parameter to me. No, I don't get refusals, I see literally invalid json, like: `{"field": ["value...}`
- koakuma-chan 1y agohttps://github.com/guidance-ai/llguidance https://github.com/guidance-ai/llguidance > 2025-05-20 LLGuidance shipped in OpenAI for JSON Schema
- mmoskal 1y agoOpenAI is using [0] LLGuidance [1]. You need to set strict:true in your request for schema validation to kick in though. [0] https://platform.openai.com/docs/guides/function-calling#lark-cfg https://platform.openai.com/docs/guides/function-calling#lar... [1] https://github.com/guidance-ai/llguidance https://github.com/guidance-ai/llguidance
- jawiggins 1y agoI don't think that parameter is an option when using pydantic schemas. class FooBar(BaseModel): foo: list[str] bar: list[int] prompt = """#Task Your job is to reply with Foo Bar, a json object with foo, a list of strings, and bar, a list of ints """ response = openai_client.chat.completions.parse( model="gpt-5-nano-2025-08-07", messages=[{"role": "system", "content": FooBar}], max_completion_tokens=4096, seed=123, response_format=CommentAnalysis, strict=True ) TypeError: Completions.parse() got an unexpected keyword argument 'strict'
- simonw 1y ago
- xpe 1y agoThis makes me wonder if there are cases where one would want the LLM to generate a syntactically invalid response (which could be identified as such) rather than guarantee syntactic validity at the potential cost of semantic accuracy.