3 ms·

Anthropic's Haiku Beats GPT-4 Turbo in Tool Use

48 points by Joschkabraun 2y ago

yinser 2y ago
This is great to see another player enter the tool use arena but anecdotally I'm having issues getting function calls to Anthropic to return proper JSON. Still feels a little fragile but hopefully they continue improving.
- Joschkabraun 2y ago
  Have you tried the new beta tool use API? In the experiments I ran there were almost no issues parsing the function call response (similar to GPT-3.5-turbo & GPT-4 turbo)
- fudoshin2596 2y ago
  Yea, new tool calling now in public beta is way better. And avoids all of that XML they had before.
airocker 2y ago
Are tools fine tuned in both or just prompt based? For a fair comparison
- Joschkabraun 2y ago
  All models got the same prompt fed which was essentially "Question: {question}". And then the API's accept the function call definition
  - airocker 2y ago
    But what if there is a bad model that fine tunes with only tools?
    - Joschkabraun 2y ago
      What do you mean with being a bad model? If the model is really good at tool use, then it will broadly useful as it needs capabilities to generate the tool definition. So, there should be some transferability.
      - ec109685 2y ago
        If the exact tools test was part of the training of the model, wouldn’t that throw off the results and not generalize?
- fudoshin2596 2y ago
  No fine tuning. Looks like he's do raw model capabilities with simple prompt. repo: https://github.com/parea-ai/tool-use-benchmark https://github.com/parea-ai/tool-use-benchmark
deleted 2y ago
[deleted]
mritchie712 2y ago
Not sure when they implemented this, but ollama now has a JSON mode [0]. Not function calling, but one of the simpler ways to get JSON in a local LLM. I'm using it with `knoopx/hermes-2-pro-mistral:7b-q8_0` and it's worked well for me so far. response = ollama.chat(model=OLLAMA_MODEL, messages=[ { 'role': 'system', 'content': system_message, }, { 'role': 'user', 'content': user_prompt, }, ], format='json', options = { #'temperature': 1.5, # very creative 'temperature': 0.0 } ) 0 - https://github.com/ollama/ollama/blob/main/docs/api.md#json-mode https://github.com/ollama/ollama/blob/main/docs/api.md#json-...
- Joschkabraun 2y ago
  Interesting. Do you have any benchmarks?
  - mritchie712 2y ago
    No benchmarks, just my anecdotal experience trying to get local LLM's to respond with JSON. The method above works for my use case nearly 100% of the time. Other things I've tried (e.g. `outlines`[0]) are really slow or don't work at all. Would love to hear what others have tried! 0 - https://github.com/outlines-dev/outlines https://github.com/outlines-dev/outlines
    - Joschkabraun 2y ago
      Ah yes. Have you tried out instructor [0] or Guidance [1]? [0]: https://github.com/jxnl/instructor/ https://github.com/jxnl/instructor/ [1]: https://github.com/guidance-ai/guidance/tree/main https://github.com/guidance-ai/guidance/tree/main
waldrews 2y ago
Now if only Claude had JSON mode.

48 points by Joschkabraun 2y ago

This is great to see another player enter the tool use arena but anecdotally I'm having issues getting function calls to Anthropic to return proper JSON. Still feels a little fragile but hopefully they continue improving.

Joschkabraun 2y ago
Have you tried the new beta tool use API? In the experiments I ran there were almost no issues parsing the function call response (similar to GPT-3.5-turbo & GPT-4 turbo)
fudoshin2596 2y ago
Yea, new tool calling now in public beta is way better. And avoids all of that XML they had before.

airocker 2y ago

Are tools fine tuned in both or just prompt based? For a fair comparison

Joschkabraun 2y ago
All models got the same prompt fed which was essentially "Question: {question}". And then the API's accept the function call definition
- airocker 2y ago
  But what if there is a bad model that fine tunes with only tools?
  - Joschkabraun 2y ago
    What do you mean with being a bad model? If the model is really good at tool use, then it will broadly useful as it needs capabilities to generate the tool definition. So, there should be some transferability.
    - ec109685 2y ago
      If the exact tools test was part of the training of the model, wouldn’t that throw off the results and not generalize?
fudoshin2596 2y ago
No fine tuning. Looks like he's do raw model capabilities with simple prompt. repo: https://github.com/parea-ai/tool-use-benchmark https://github.com/parea-ai/tool-use-benchmark

deleted 2y ago

[deleted]

mritchie712 2y ago

Not sure when they implemented this, but ollama now has a JSON mode [0]. Not function calling, but one of the simpler ways to get JSON in a local LLM. I'm using it with `knoopx/hermes-2-pro-mistral:7b-q8_0` and it's worked well for me so far. response = ollama.chat(model=OLLAMA_MODEL, messages=[ { 'role': 'system', 'content': system_message, }, { 'role': 'user', 'content': user_prompt, }, ], format='json', options = { #'temperature': 1.5, # very creative 'temperature': 0.0 } ) 0 - https://github.com/ollama/ollama/blob/main/docs/api.md#json-mode https://github.com/ollama/ollama/blob/main/docs/api.md#json-...

Joschkabraun 2y ago
Interesting. Do you have any benchmarks?
- mritchie712 2y ago
  No benchmarks, just my anecdotal experience trying to get local LLM's to respond with JSON. The method above works for my use case nearly 100% of the time. Other things I've tried (e.g. `outlines`[0]) are really slow or don't work at all. Would love to hear what others have tried! 0 - https://github.com/outlines-dev/outlines https://github.com/outlines-dev/outlines
  - Joschkabraun 2y ago
    Ah yes. Have you tried out instructor [0] or Guidance [1]? [0]: https://github.com/jxnl/instructor/ https://github.com/jxnl/instructor/ [1]: https://github.com/guidance-ai/guidance/tree/main https://github.com/guidance-ai/guidance/tree/main

waldrews 2y ago

Now if only Claude had JSON mode.