4 ms·
Big proponent of guaranteed outputs for LLMs. I wrote a library awhile back (gpt-json) that did something similar by querying the OpenAI API. At the end of the
by icyfox 3y ago
Big proponent of guaranteed outputs for LLMs. I wrote a library awhile back (gpt-json) that did something similar by querying the OpenAI API. At the end of the day though while their responses are _highly likely_ to be valid JSON they're not guaranteed. There's only so much that can be done with remote calls to their model's black box.
The future here really lies in compiling down context free grammars. They let you model json, yml, csv, and other programming languages as finite state machines that can force LLM transitions. They end up being pretty magical: you can force value typing, enums, and syntax validation of multivariate payloads. For use in data pipelines they can't be beat.
I did some experiments a few weeks ago on training models to generate these formats explicitly with jsonformers/outlines. Finetuning in the right format is still important to maximize output. You can end up seeing a 7% lift if you finetune explicitly for your desired format. [^1] At inference time the CFGs will constrain your model to do what it's actually intended to.
[^1]: https://freeman.vc/notes/constraining-llm-outputs https://freeman.vc/notes/constraining-llm-outputs
- edunteman 3y agoit seems this is in the context of "extraction" where all of the data is already present in the input text, and all that's needed is the reformatting. This is something we've been wrestling with (even today): is the role of fructose and/or our eventual formatting model to provide both intelligence + formatting (such as generating novel data, on the fly, constrained to structure), or just formatting (anything-to-json). Not sure what the answer is and not expecting one, we're just realizing the clear split between needs of users running extraction vs more generative/creative tasks.
- KTibow 3y agoOpenAI lets you force valid JSON now https://platform.openai.com/docs/guides/text-generation/json-mode https://platform.openai.com/docs/guides/text-generation/json...
- edunteman 3y agovalid json, yes, but not a specific json schema (yet, who knows, maybe they ship schema support, I'm surprised they haven't)
- disconcision 3y agowe don't need to limit ourselves to context-free, either. its possible to enforce scope as well, and even force per-token type correctness, as least for somewhat syntactically well-behaved languages that use local type inference.