Integrating LLMs and guardrailing the API outputs used to be an interview question. It's one of the ways you could tell if someone actually built a sufficiently

Integrating LLMs and guardrailing the API outputs used to be an interview question. It's one of the ways you could tell if someone actually built a sufficiently complex AI tool in production. 1. LLMs hallucinate and often forget to close a bracket or leave a field out. This still happens in JSON mode like Gemini when it's a feature. 2. JSON formatting uses a lot of unnecessary tokens. Comma, quotes, brackets, etc. 3. Extra tokens also mean extra "cognitive effort" for the LLMs. We changed to YAML from JSON and saw a 30% or so increase in output quality back with GPT-3.5. 4. The above can be fixed with more and more training, but why train for REST when you can build something better for it?