4 ms·

> All models can handle correcting choosing a tool from 250+ tools provided the user query contains all necessary parameters for the intended tool with >90% acc

> All models can handle correcting choosing a tool from 250+ tools provided the user query contains all necessary parameters for the intended tool with >90% accuracy. This is pretty exciting news for everybody working with agentic systems. OpenAI has way lower recall. I'm now migrating from GPT function calls to Claude tools and will report back on the evaluation results.

vorticalbox 3y ago
I thought this too where it will usually pick stuff listed first rather than a more suitable tool down in the list. Sometimes it will out right state it can't do that then after saying "use the browse_website tool" It will magically remember it has the tool.
iAkashPaul 3y ago
You should do the new HF TGI server, it has both grammar & tool support now. Works fabulously with Mistral Instruct & Mixtral Instruct.
- Takennickname 3y ago
  Whats grammar support?
  - iAkashPaul 3y ago
    It's basically guidelines to adhere in the output. Ex. Your prompt asks for a summary, the LLM won't necessarily just spit the summary, there might be phrases like here's the summary - then the actual summary or even basic JSON key/values can get messed up. So grammar allows for defining the expected output in a variety of ways, some via Pydantic class definitions, Microsoft's Guidance, Outlines along with llama.cpp's grammar are attempts at making structured output reliable. Most of langchain is basically a specific prompt with exception handling for missing fields.
    - dartos 3y ago
      Even better than guidelines! Grammars perfectly restrict LLM token output. They’re hard grammar rules.
  - sp332 3y ago
    You can give it a formal definition for valid JSON, and it will only generate output that matches.
mmoustafa 3y ago
Claude's [new] tool usage is pretty good. Unlike with GPT-4 where I had to really minimize the context and descriptions for each tool, Claude Opus does better when provided more details and context for each tool, much more nuanced. I'm now using it with 9 different tools for https://olly.bot https://olly.bot and it hits the nail on the head about 8/10 times. Anthropic says it can handle 250+ tools with 90% accuracy [1], but anecdotally from my production usage in the last 24 hours that seems a little too optimistic. Annnd, it also comes with a few idiosyncracies like sometimes spitting out <thinking> or <answer> blocks, and has more constraints on the messages field, so don't expect a drop-in replacement for OpenAI. [1] https://docs.anthropic.com/claude/docs/tool-use https://docs.anthropic.com/claude/docs/tool-use
- cpursley 3y ago
  Olly is really neat, I just set up a chat with it. How did you architect the web search (tools?) if you don't mind sharing?

vorticalbox 3y ago

I thought this too where it will usually pick stuff listed first rather than a more suitable tool down in the list. Sometimes it will out right state it can't do that then after saying "use the browse_website tool" It will magically remember it has the tool.

iAkashPaul 3y ago

You should do the new HF TGI server, it has both grammar & tool support now. Works fabulously with Mistral Instruct & Mixtral Instruct.

Takennickname 3y ago
Whats grammar support?
- iAkashPaul 3y ago
  It's basically guidelines to adhere in the output. Ex. Your prompt asks for a summary, the LLM won't necessarily just spit the summary, there might be phrases like here's the summary - then the actual summary or even basic JSON key/values can get messed up. So grammar allows for defining the expected output in a variety of ways, some via Pydantic class definitions, Microsoft's Guidance, Outlines along with llama.cpp's grammar are attempts at making structured output reliable. Most of langchain is basically a specific prompt with exception handling for missing fields.
  - dartos 3y ago
    Even better than guidelines! Grammars perfectly restrict LLM token output. They’re hard grammar rules.
- sp332 3y ago
  You can give it a formal definition for valid JSON, and it will only generate output that matches.

mmoustafa 3y ago

Claude's [new] tool usage is pretty good. Unlike with GPT-4 where I had to really minimize the context and descriptions for each tool, Claude Opus does better when provided more details and context for each tool, much more nuanced. I'm now using it with 9 different tools for https://olly.bot https://olly.bot and it hits the nail on the head about 8/10 times. Anthropic says it can handle 250+ tools with 90% accuracy [1], but anecdotally from my production usage in the last 24 hours that seems a little too optimistic. Annnd, it also comes with a few idiosyncracies like sometimes spitting out <thinking> or <answer> blocks, and has more constraints on the messages field, so don't expect a drop-in replacement for OpenAI. [1] https://docs.anthropic.com/claude/docs/tool-use https://docs.anthropic.com/claude/docs/tool-use

cpursley 3y ago
Olly is really neat, I just set up a chat with it. How did you architect the web search (tools?) if you don't mind sharing?