3 ms·

You should do the new HF TGI server, it has both grammar & tool support now. Works fabulously with Mistral Instruct & Mixtral Instruct.

• Takennickname 3y ago
  Whats grammar support?
  • iAkashPaul 3y ago
    It's basically guidelines to adhere in the output. Ex. Your prompt asks for a summary, the LLM won't necessarily just spit the summary, there might be phrases like here's the summary - then the actual summary or even basic JSON key/values can get messed up. So grammar allows for defining the expected output in a variety of ways, some via Pydantic class definitions, Microsoft's Guidance, Outlines along with llama.cpp's grammar are attempts at making structured output reliable. Most of langchain is basically a specific prompt with exception handling for missing fields.
    • dartos 3y ago
      Even better than guidelines! Grammars perfectly restrict LLM token output. They’re hard grammar rules.
  • sp332 3y ago
    You can give it a formal definition for valid JSON, and it will only generate output that matches.