3 ms·

Can anyone comment on how well this does at coercing json output vs OpenAI function calling?

• verdverm 3y ago
  This is just a different way to write prompts, it allows some interleaving of calls to the API so you can build things up, write a conversation as a single file, with conventions around the text to send to the LLM. I would not expect it to make a difference in your current applications. Getting JSON is all about the model, training, and prompt, in that order If you are looking for low-hanging fruit to improve your JSON responses from LLMs, fine-tuning will likely get you the most bang for your buck. Start from a coding model like codellama, code-bison, or starcoder
  • startupsfail 3y ago
    For the local model it forces valid json structure and formatting tokens are being produced by code rather than generated by an LLM.
    • verdverm 3y ago
      sounds like post-processing made out to be something more? everyone is doing this, it's just part of the pipeline, certainly nothing innovative on that front in guidance
      • mmoskal 3y ago
        It updates token logits (probabilities) after every token before sampling. I don't think this is very common yet.
        • newhouseb 3y ago
          Right, there are many folks (dozens of us!) yelling about logit processors and building them into various frameworks. The mostly widely accessible form of this is probably BNF grammar biasing in llama.cpp: https://github.com/ggerganov/llama.cpp/blob/master/grammars/README.md https://github.com/ggerganov/llama.cpp/blob/master/grammars/...
        • verdverm 3y ago
          anecdotal counter evidence, I've seen multiple projects / papers manipulating the logits, it's a very common thing to think of doing now to improve performance (by eliminating bad options from consideration)
        • Der_Einzige 3y ago
          Still rare, but I wrote a whole paper last year about what happens when you use this functionality (a lot, including defeating any kind of RLHF!) https://aclanthology.org/2022.cai-1.2.pdf https://aclanthology.org/2022.cai-1.2.pdf
• bugglebeetle 3y ago
  OpenAI function calling + JSON schema is dead simple and has never failed for me, where as I had a bunch of errors with guidance when trying to do things like nested, repeating values.
  • verdverm 3y ago
    Yeah, my problem with this is that you have to buy into their way of interacting with and calling an LLM. Seems more like Handcuffs than Guidance to me