3 ms·

This is great! AIUI, llama.cpp does support tools, but I haven't figured out yet what to do to make llm use it. Is there anything I can put into extra-openai-m

by lynx97 1y ago

This is great! AIUI, llama.cpp does support tools, but I haven't figured out yet what to do to make llm use it. Is there anything I can put into extra-openai-models.yaml to make this work?

simonw 1y ago
That's likely a change that needs to be made to either https://github.com/simonw/llm-gguf https://github.com/simonw/llm-gguf or https://github.com/simonw/llm-llama-server https://github.com/simonw/llm-llama-server ... OK, I got the second one working! brew install llama.cpp llama-server --jinja -hf unsloth/gemma-3-4b-it-GGUF:Q4_K_XL Then in another window: llm install llm-llama-server llm -m llama-server-tools -T llm_time 'what time is it?' --td Wrote it up here: https://simonwillison.net/2025/May/28/llama-server-tools/ https://simonwillison.net/2025/May/28/llama-server-tools/
- johnisgood 1y ago
  Simon, at this point you have a lot of LLM-related tools, and I am not sure which one is outdated, which one is the newest, fanciest stuff, which one is the one that one should use (and when), and so forth. Is there a blog post / article that addresses this?
  - simonw 1y ago
    For my stuff it's basically just the latest https://llm.datasette.io https://llm.datasette.io and its collection of plugins. If you're interested in what I recommend generally that changes a lot, but my most recent piece about that is here: https://simonwillison.net/2025/May/15/building-on-llms/ https://simonwillison.net/2025/May/15/building-on-llms/
- lynx97 1y ago
  Great, this works here. I wonder, with extra-openai-models.yaml I was able to set the api_base and vision/audio: true. How do I do this with the llama-server-tools plugin? vision works, but llm refuses to attach audio because it thinks the model does not support audio /which it does). EDIT: I think I just found what I want. There is no need for the plugin, extra-openai-models.yaml just needs "supports_tools: true" and "can_stream: false".

by lynx97 1y ago

This is great! AIUI, llama.cpp does support tools, but I haven't figured out yet what to do to make llm use it. Is there anything I can put into extra-openai-models.yaml to make this work?

simonw 1y ago

That's likely a change that needs to be made to either https://github.com/simonw/llm-gguf https://github.com/simonw/llm-gguf or https://github.com/simonw/llm-llama-server https://github.com/simonw/llm-llama-server ... OK, I got the second one working! brew install llama.cpp llama-server --jinja -hf unsloth/gemma-3-4b-it-GGUF:Q4_K_XL Then in another window: llm install llm-llama-server llm -m llama-server-tools -T llm_time 'what time is it?' --td Wrote it up here: https://simonwillison.net/2025/May/28/llama-server-tools/ https://simonwillison.net/2025/May/28/llama-server-tools/

johnisgood 1y ago
Simon, at this point you have a lot of LLM-related tools, and I am not sure which one is outdated, which one is the newest, fanciest stuff, which one is the one that one should use (and when), and so forth. Is there a blog post / article that addresses this?
- simonw 1y ago
  For my stuff it's basically just the latest https://llm.datasette.io https://llm.datasette.io and its collection of plugins. If you're interested in what I recommend generally that changes a lot, but my most recent piece about that is here: https://simonwillison.net/2025/May/15/building-on-llms/ https://simonwillison.net/2025/May/15/building-on-llms/
lynx97 1y ago
Great, this works here. I wonder, with extra-openai-models.yaml I was able to set the api_base and vision/audio: true. How do I do this with the llama-server-tools plugin? vision works, but llm refuses to attach audio because it thinks the model does not support audio /which it does). EDIT: I think I just found what I want. There is no need for the plugin, extra-openai-models.yaml just needs "supports_tools: true" and "can_stream: false".