2 ms·
I really hope OpenAI add something like this to their endpoints soon. Being able to pass up some kind of grammar (a regular expression, or a JSON schema, or so
by simonw 3y ago
I really hope OpenAI add something like this to their endpoints soon.
Being able to pass up some kind of grammar (a regular expression, or a JSON schema, or some other format) and have this trick run during their token sampling process to ensure the output was compliant would be incredibly useful.
- joshuanapoli 3y agoIsn't the Function Calling feature meant for this purpose? It guides the LLM to output according to the given schema. The name of the feature is a little misleading. https://platform.openai.com/docs/guides/gpt/function-calling https://platform.openai.com/docs/guides/gpt/function-calling
- simonw 3y agoSurprisingly the function calling mechanism doesn't appear to use this trick - apparently it's still possible to get the wrong JSON structure back from it occasionally.
- tornato7 3y agoFunction Calling is fine-tuned to a certain output format, but it very often strays from that format. My function-calling-handling code has a mess of edge case handlers that catch when GPT-4 is calling functions incorrectly.
- M4v3R 3y agoIt’s not though, they even say it in their docs that sending a schema does not guarantee that the model will actually adhere to the scheme or even produce valid JSON
- potatoman22 3y agoThey recently added logit biases, so that's a start.
- remilouf 3y agoIt's limited to 300 logit biases at a time. Knowing GPT4's vocabulary is ~100k tokens it's not nearly enough to get reliable guided generation. Although it could work in some cases, and another advantage of this work is that we can determine that before generating.