3 ms·

Why would someone want `strict` to be anything other than `true`?

davidkunz 2y ago
Maybe if you can't precisely model your structure with (OpenAI's subset of) JSON schema.
ComputerGuru 2y ago
There are many reasons, though I am not sure which they had in mind. One thing is that LLMs in general tend to do better when they can be more verbose in their output and sort of “think aloud” to reach an answer. Insisting on strict output format would rob it of the benefits (because it doesn’t just not emit but completely skips those stages, or else you’d be paying for those elided output tokens).
- wewtyflakes 2y ago
  But then why would someone specify that the response has to be in a given JSON schema (by presence of the schema itself), but then also not care if it is actually using that schema (by specifying `strict` as `false`)? That is the use-case I can't wrap my head around.
  - ComputerGuru 2y ago
    Ah, good point. It honestly feels/looks like a hack to flip a switch on their end and they’re just bubbling it back up to the user.
tedsanders 2y ago
We didn't cover this in the announcement post, but there are a few reasons: - The first request with each JSON schema will be slow, as we need to preprocess the JSON schema into a context-free grammar. If you don't want that latency hit (e.g., you're prototyping, or have a use case that uses variable one-off schemas), then you might prefer "strict": false - You might have a schema that isn't covered by our subset of JSON schema. (To keep performance fast, we don't support some more complex/long-tail features.) - In JSON mode and Structured Outputs, failures are rarer but more catastrophic. If the model gets too confused, it can get stuck in loops where it just prints technically valid output forever without ever closing the object. In these cases, you can end up waiting a minute for the request to hit the max_token limit, and you also have to pay for all those useless tokens. So if you have a really tricky schema, and you'd rather get frequent failures back quickly instead of infrequent failures back slowly, you might also want "strict": false But in 99% of cases, you'll want "strict": true.
- solaxun 2y ago
  On the latency of the first request - How is the CFG cached? Is it done at the API Key + schema level? Meaning that for a given API key, the latency penalty for a new schema is only paid one time, regardless of how far apart requests are? Or is cached with less duration, e.g. each session, conversation thread, etc?

Maybe if you can't precisely model your structure with (OpenAI's subset of) JSON schema.

There are many reasons, though I am not sure which they had in mind. One thing is that LLMs in general tend to do better when they can be more verbose in their output and sort of “think aloud” to reach an answer. Insisting on strict output format would rob it of the benefits (because it doesn’t just not emit but completely skips those stages, or else you’d be paying for those elided output tokens).

wewtyflakes 2y ago
But then why would someone specify that the response has to be in a given JSON schema (by presence of the schema itself), but then also not care if it is actually using that schema (by specifying `strict` as `false`)? That is the use-case I can't wrap my head around.
- ComputerGuru 2y ago
  Ah, good point. It honestly feels/looks like a hack to flip a switch on their end and they’re just bubbling it back up to the user.

tedsanders 2y ago

We didn't cover this in the announcement post, but there are a few reasons: - The first request with each JSON schema will be slow, as we need to preprocess the JSON schema into a context-free grammar. If you don't want that latency hit (e.g., you're prototyping, or have a use case that uses variable one-off schemas), then you might prefer "strict": false - You might have a schema that isn't covered by our subset of JSON schema. (To keep performance fast, we don't support some more complex/long-tail features.) - In JSON mode and Structured Outputs, failures are rarer but more catastrophic. If the model gets too confused, it can get stuck in loops where it just prints technically valid output forever without ever closing the object. In these cases, you can end up waiting a minute for the request to hit the max_token limit, and you also have to pay for all those useless tokens. So if you have a really tricky schema, and you'd rather get frequent failures back quickly instead of infrequent failures back slowly, you might also want "strict": false But in 99% of cases, you'll want "strict": true.

solaxun 2y ago
On the latency of the first request - How is the CFG cached? Is it done at the API Key + schema level? Meaning that for a given API key, the latency penalty for a new schema is only paid one time, regardless of how far apart requests are? Or is cached with less duration, e.g. each session, conversation thread, etc?