3 ms·

How do you guarantee output structure? Does it ever fail to conform?

• edunteman 3y ago
  It's not 100% yet. Route to that: 1. Clientside, retry strategy on failed parse. Not yet implemented, we throw an exception on parse fail right now, but soon to be implemented. Not ideal because of token burn and latency, but the best quick solution. 2. For the custom model we're building, we use strict grammar definitions to bias outputs toward the needed structure (or if there is only one structurally correct token, outright skip the generation of that token and insert it directly). I've been impressed at how well gpt-4 does with the default prompt template we use. Even better if you enable the chain_of_thought flavor.
  • bagels 3y ago
    Basically my experience with homegrown. If you give it a typescript template for outputs, and a ton of prose to tell it how to respond, you usually get the right responses.