3 ms·

I'm in the middle of moving my app to openAI structured output. Could you please explain what you mean by "which everyone moved on from"?

• slickytail 8d ago
  Structured decoding limits next-token probabilities to ensure valid JSON. The main issue is that if the model puts a substantial probability on an invalid token, then it was already confused, and in that case, you don't actually want whatever the next-most-likely valid token is: even if it's syntactically valid, it's likely semantically erroneous.
  • rhodysurf 8d ago
    So whats the alternative? Letting it codegen a file and the piping that? What a worse workflow
    • qeternity 7d ago
      Alternative is to fail the validation and regenerate the request, but I have no idea what this person is talking about. It is absolutely not the case that everyone has moved on from structured generation. Absurd claim.
  • slices 8d ago
    sounds like an argument in favor of structured output, is that right?
  • qeternity 7d ago
    Not sure why you think everyone has moved on from this. Valid JSON is only one aspect. It almost certainly has far more uptake today than ca. Sonnet 3.7. Grammar engines can enforce a lot of other things too. Models are actually really good at emitting valid JSON, but complex schemas effectively require it.