4 ms·
Interesting paper, but their reason for dismissing constrained decoding methods seems to be that they want to academically study the in-context setting. For pr
by FlyingLawnmower 2y ago
Interesting paper, but their reason for dismissing constrained decoding methods seems to be that they want to academically study the in-context setting.
For practitioners, using a framework like Guidance which forces the models to write valid JSON as they generate text solves this trivially (https://github.com/guidance-ai/guidance https://github.com/guidance-ai/guidance)
For json in particular these frameworks have functions that take in json schemas or pydantic schemas https://guidance.readthedocs.io/en/latest/generated/guidance.json.html#guidance-json https://guidance.readthedocs.io/en/latest/generated/guidance...
- myprotegeai 2y agoAnd OpenAI also has Structured Outputs[1] that has the same effect as Guidance. I use it to safely deserialize remote function calls based on a jsonschema[2]. It works very well. 1. https://openai.com/index/introducing-structured-outputs-in-the-api/ https://openai.com/index/introducing-structured-outputs-in-t... 2. https://github.com/amoffat/manifest https://github.com/amoffat/manifest
- imtringued 2y agoTo be fair, they first build a benchmark which they call "StructuredRAG" and it doesn't make sense to run constrained decoding against a benchmark, because it would always get you a 100% success chance. Once they have a benchmark, they try to figure out whether it is possible to prompt engineer your way to a 100% success rate and by using ORPO to generate the prompt, they did achieve that 100% success rate without relying on constrained decoding.