4 ms·
I mean, the constrained output mode also doesn’t hallucinate in this sense.
by seizethecheese 12d ago
I mean, the constrained output mode also doesn’t hallucinate in this sense.
- dropofwill 11d agoThey do actually admit that about constrained decoding somewhere in the docs. They argue it’s useless in practice because when the constraints actually kick in it harms the output too much and that it’s better to just error and retry in those cases. That does align with my experience, though we’re not using anything close to frontier for these sort of tasks. I am interested if it can actually improve on that. As an engineer i like the elegance of guaranteed output, but the retry works pretty well in practice.