5 ms·
TGI just integrated Guidance in 1.4.3, that by itself can support both grammar/JSON/Pydantic & tool invocation/function calling. Langchain & Llamaindex plus Fr
by iAkashPaul 3y ago
TGI just integrated Guidance in 1.4.3, that by itself can support both grammar/JSON/Pydantic & tool invocation/function calling.
Langchain & Llamaindex plus Fructose really need to skip the structure adherence work & move to chunking/KG generation since that's the next pain point to tackle.
- memothon 3y agoWhat is TGI? Is that the huggingface Text Generation Inference project on GitHub?
- edunteman 3y agoYes. TGI is Huggingface's version of LLVM (some nuance, of course). LLVM also launched grammar support recently too, so we'll be looking into it.
- memothon 3y agoAll of these acronyms are so confusing. I'm assuming LLVM isn't the compiler tool, but searching "LLVM ai" doesn't give me any good results.
- reliableturing 3y agoThey probably meant vLLM https://docs.vllm.ai/en/latest/ https://docs.vllm.ai/en/latest/
- edunteman 3y agoah shoot, yes I meant vLLM, sorry for the confusion, lots of comments to reply to :)
- Terretta 3y agoWhy do you have Guidance in caps? https://github.com/guidance-ai/guidance https://github.com/guidance-ai/guidance or ... https://huggingface.co/docs/text-generation-inference/conceptual/guidance https://huggingface.co/docs/text-generation-inference/concep... or ... ? A quick glance through these, they don't seem yet to leverage json_object on OpenAI with the word JSON in the prompt, which works wonders with the 0125 models.