3 ms·

How sure are you that OpenAI is using that? I would have suspected it too, but I’ve been struggling with OpenAI returning syntactically invalid JSON when provi

How sure are you that OpenAI is using that? I would have suspected it too, but I’ve been struggling with OpenAI returning syntactically invalid JSON when provided with a simple pydantic class (a list of strings), which shouldn’t be possible unless they have a glaring error in their grammar.

gradys 11mo ago
You might be using JSON mode, which doesn’t guarantee a schema will be followed, or structured outputs not in strict mode. It is possible to get the property that the response is either a valid instance of the schema or an error (eg for refusal)
- jawiggins 11mo ago
  How do you activate strict mode when using pydantic schemas? It doesn't look like that is a valid parameter to me. No, I don't get refusals, I see literally invalid json, like: `{"field": ["value...}`
koakuma-chan 11mo ago
https://github.com/guidance-ai/llguidance https://github.com/guidance-ai/llguidance > 2025-05-20 LLGuidance shipped in OpenAI for JSON Schema
mmoskal 11mo ago
OpenAI is using [0] LLGuidance [1]. You need to set strict:true in your request for schema validation to kick in though. [0] https://platform.openai.com/docs/guides/function-calling#lark-cfg https://platform.openai.com/docs/guides/function-calling#lar... [1] https://github.com/guidance-ai/llguidance https://github.com/guidance-ai/llguidance
- jawiggins 11mo ago
  I don't think that parameter is an option when using pydantic schemas. class FooBar(BaseModel): foo: list[str] bar: list[int] prompt = """#Task Your job is to reply with Foo Bar, a json object with foo, a list of strings, and bar, a list of ints """ response = openai_client.chat.completions.parse( model="gpt-5-nano-2025-08-07", messages=[{"role": "system", "content": FooBar}], max_completion_tokens=4096, seed=123, response_format=CommentAnalysis, strict=True ) TypeError: Completions.parse() got an unexpected keyword argument 'strict'
simonw 11mo ago
You have to explicitly opt into it by passing strict=True https://platform.openai.com/docs/guides/structured-outputs/supported-schemas https://platform.openai.com/docs/guides/structured-outputs/s...
- jawiggins 11mo ago
  Are you able to use `strict=True` when using pydantic models? It doesn't seem to be valid for me. I think that only works for json schemas. class FooBar(BaseModel): foo: list[str] bar: list[int] prompt = """#Task Your job is to reply with Foo Bar, a json object with foo, a list of strings, and bar, a list of ints """ response = openai_client.chat.completions.parse( model="gpt-5-nano-2025-08-07", messages=[{"role": "system", "content": FooBar}], max_completion_tokens=4096, seed=123, response_format=CommentAnalysis, strict=True ) > TypeError: Completions.parse() got an unexpected keyword argument 'strict'

gradys 11mo ago

You might be using JSON mode, which doesn’t guarantee a schema will be followed, or structured outputs not in strict mode. It is possible to get the property that the response is either a valid instance of the schema or an error (eg for refusal)

jawiggins 11mo ago
How do you activate strict mode when using pydantic schemas? It doesn't look like that is a valid parameter to me. No, I don't get refusals, I see literally invalid json, like: `{"field": ["value...}`

koakuma-chan 11mo ago

https://github.com/guidance-ai/llguidance https://github.com/guidance-ai/llguidance > 2025-05-20 LLGuidance shipped in OpenAI for JSON Schema

mmoskal 11mo ago

OpenAI is using [0] LLGuidance [1]. You need to set strict:true in your request for schema validation to kick in though. [0] https://platform.openai.com/docs/guides/function-calling#lark-cfg https://platform.openai.com/docs/guides/function-calling#lar... [1] https://github.com/guidance-ai/llguidance https://github.com/guidance-ai/llguidance

jawiggins 11mo ago
I don't think that parameter is an option when using pydantic schemas. class FooBar(BaseModel): foo: list[str] bar: list[int] prompt = """#Task Your job is to reply with Foo Bar, a json object with foo, a list of strings, and bar, a list of ints """ response = openai_client.chat.completions.parse( model="gpt-5-nano-2025-08-07", messages=[{"role": "system", "content": FooBar}], max_completion_tokens=4096, seed=123, response_format=CommentAnalysis, strict=True ) TypeError: Completions.parse() got an unexpected keyword argument 'strict'

simonw 11mo ago

You have to explicitly opt into it by passing strict=True https://platform.openai.com/docs/guides/structured-outputs/supported-schemas https://platform.openai.com/docs/guides/structured-outputs/s...

jawiggins 11mo ago
Are you able to use `strict=True` when using pydantic models? It doesn't seem to be valid for me. I think that only works for json schemas. class FooBar(BaseModel): foo: list[str] bar: list[int] prompt = """#Task Your job is to reply with Foo Bar, a json object with foo, a list of strings, and bar, a list of ints """ response = openai_client.chat.completions.parse( model="gpt-5-nano-2025-08-07", messages=[{"role": "system", "content": FooBar}], max_completion_tokens=4096, seed=123, response_format=CommentAnalysis, strict=True ) > TypeError: Completions.parse() got an unexpected keyword argument 'strict'