7 ms·

Old news by now, but you might not be aware that llama-server can do multi-model for a while now, Meaning that you (and by that I mean your AI agent that has r

by hypfer 2mo ago

Old news by now, but you might not be aware that llama-server can do multi-model for a while now, Meaning that you (and by that I mean your AI agent that has read the llama.cpp code) can write an ini file pointing to your models with parameters optimized for the specific model on your specific hardware. (Optimized by you through testing. Not that AI) Then, any api client can just select a model and the system does the right thing. It's great software. It just works. __ You just need to ignore the cargo culting commandline options on social media. But you should be listening to the devs. Have you already enabled ngram-mod (or rather just spec-default)? It is practically free.

mrighele 2mo ago
> (Optimized by you through testing. Not that AI) Why not optimized by AI through testing ? Give it a test set to work on and let it loose.
- LoganDark 2mo ago
  AI doesn't necessarily know what feels like a good tradeoff to you. I'm sure it could help guide you though.
  - hypfer 2mo ago
    ^ This. Intent is the answer and AI has none.
  - Zetaphor 2mo ago
    This is all quantifiable. I regularly have my model run benchmarks against all the config permutations and then choose the best based on my criteria, which typically boil down to trading prefill and decode times
    - LoganDark 2mo ago
      What model has to trade between those? I have both. You just have different, independently-optimized forward passes for each.
      - Zetaphor 2mo ago
        I'm running on Strix Halo so memory bandwidth is my constraint. In that example I'm describing the choice between using ROCm or Vulkan. I have a llama-swap config that can call different instances of llama-server running a toolbox with either runtime.
        LoganDark 2mo ago
        Memory bandwidth is my constraint too (M4 Max) but prefill and single-token decode don't run at the same time. It's best to use batched prefill so you can benefit from processing multiple tokens with a single pass through the model weights.
jwr 2mo ago
Caveat: this kind-of-works. There are plenty of subtle bugs around this. For example, it's hard to have a default model, and clients need to select the model explicitly in every API request, which makes it difficult to work with several models and multiple clients.
- cptskippy 2mo ago
  There are tools like llama-swap that make this pretty bullet proof and offer an OAI style API and compatibility for local Agents. I've been running flavors of OpenClaw and Hermes exclusively on local models hosted this way. https://github.com/mostlygeek/llama-swap https://github.com/mostlygeek/llama-swap
  - tredre3 2mo ago
    If you can get away with it, you really should stick to llama-server alone. When they added support the ability to load multiple models + an idle timeout to unload them, it finally did it for me. Adding llama-swap adds a fair amount of complexity that is unpleasant to debug and its documentation isn't very good (the only real doc is the sample config file).
    - cptskippy 2mo ago
      The native router matrix for llama.cpp is still pretty immature (though evolving quickly) and I plan to switch once it reaches feature parity. Personally, I don't see the value in idle timeouts or time based eviction. It doesn't improve model load performance but does mean you're more likely to incur a load penalty for any request. I generally use the same two models for my agentic workloads and they sit comfortably next to one another in the 32GB of VRAM on my GPU. If I need to load a larger model, llama-swap can easily eject both models if necessary.
redmoonx 2mo ago
[dead]
timschumi 2mo ago
Is that with vanilla llama.cpp or with the third-party llama-swap manager? Last time I checked llama-swap was still the go-to solution, although I admit I haven't looked into it further.
- NorwegianDude 2mo ago
  llama.cpp has router mode these days, swapping out models for you, you no longer need llama-swap.
  - dangoljames 2mo ago
    I use it on the daily. Haven't exactly missed ollama.
  - cptskippy 2mo ago
    The router mode and matrix routing in llama.cpp is still early days and it can't easily juggle multiple models as easily as llama-swap so there's still benefits if you're using 24-32GB cards that can run multiple models simultaneously. The gulf is quickly shrinking though and it seems like the need for llama-swap will disappear soon.
- hypfer 2mo ago
  If you revisit my comment and pay attention to the opener: > but you might not be aware that llama-server can do multi-model for a while now you will see that the sentence structure clearly implies both a change compared with a prior state and also lack of any third-party thing. So the answer to the question has already been encoded as text available. _ I can see the desire for explicit validation though. For that, I would propose a sentence structure like > Oh cool! That means that llama-swap is now superseded/no longer needed? That shows that you've read and understand the message, gives you the double-check and might on top spark a conversation about how these solutions compare. Plus that if the guy you're commenting too has spoken nonsense, they need to backpedal.
  - emulio 2mo ago
deleted 2mo ago
[deleted]
toyg 2mo ago
> any api client can just select a model and the system does the right thing My machine can only load one model at a time. The loading/unloading times just don't seem worth the switch. I tend to use qwen3.6 for anything and that's it. Then again, I am a simple coder.

by hypfer 2mo ago

mrighele 2mo ago

> (Optimized by you through testing. Not that AI) Why not optimized by AI through testing ? Give it a test set to work on and let it loose.

LoganDark 2mo ago
AI doesn't necessarily know what feels like a good tradeoff to you. I'm sure it could help guide you though.
- hypfer 2mo ago
  ^ This. Intent is the answer and AI has none.
- Zetaphor 2mo ago
  This is all quantifiable. I regularly have my model run benchmarks against all the config permutations and then choose the best based on my criteria, which typically boil down to trading prefill and decode times
  - LoganDark 2mo ago
    What model has to trade between those? I have both. You just have different, independently-optimized forward passes for each.
    - Zetaphor 2mo ago
      I'm running on Strix Halo so memory bandwidth is my constraint. In that example I'm describing the choice between using ROCm or Vulkan. I have a llama-swap config that can call different instances of llama-server running a toolbox with either runtime.
      - LoganDark 2mo ago
        Memory bandwidth is my constraint too (M4 Max) but prefill and single-token decode don't run at the same time. It's best to use batched prefill so you can benefit from processing multiple tokens with a single pass through the model weights.

jwr 2mo ago

Caveat: this kind-of-works. There are plenty of subtle bugs around this. For example, it's hard to have a default model, and clients need to select the model explicitly in every API request, which makes it difficult to work with several models and multiple clients.

cptskippy 2mo ago
There are tools like llama-swap that make this pretty bullet proof and offer an OAI style API and compatibility for local Agents. I've been running flavors of OpenClaw and Hermes exclusively on local models hosted this way. https://github.com/mostlygeek/llama-swap https://github.com/mostlygeek/llama-swap
- tredre3 2mo ago
  If you can get away with it, you really should stick to llama-server alone. When they added support the ability to load multiple models + an idle timeout to unload them, it finally did it for me. Adding llama-swap adds a fair amount of complexity that is unpleasant to debug and its documentation isn't very good (the only real doc is the sample config file).
  - cptskippy 2mo ago
    The native router matrix for llama.cpp is still pretty immature (though evolving quickly) and I plan to switch once it reaches feature parity. Personally, I don't see the value in idle timeouts or time based eviction. It doesn't improve model load performance but does mean you're more likely to incur a load penalty for any request. I generally use the same two models for my agentic workloads and they sit comfortably next to one another in the 32GB of VRAM on my GPU. If I need to load a larger model, llama-swap can easily eject both models if necessary.

redmoonx 2mo ago

[dead]

timschumi 2mo ago

Is that with vanilla llama.cpp or with the third-party llama-swap manager? Last time I checked llama-swap was still the go-to solution, although I admit I haven't looked into it further.

NorwegianDude 2mo ago
llama.cpp has router mode these days, swapping out models for you, you no longer need llama-swap.
- dangoljames 2mo ago
  I use it on the daily. Haven't exactly missed ollama.
- cptskippy 2mo ago
  The router mode and matrix routing in llama.cpp is still early days and it can't easily juggle multiple models as easily as llama-swap so there's still benefits if you're using 24-32GB cards that can run multiple models simultaneously. The gulf is quickly shrinking though and it seems like the need for llama-swap will disappear soon.
hypfer 2mo ago
If you revisit my comment and pay attention to the opener: > but you might not be aware that llama-server can do multi-model for a while now you will see that the sentence structure clearly implies both a change compared with a prior state and also lack of any third-party thing. So the answer to the question has already been encoded as text available. _ I can see the desire for explicit validation though. For that, I would propose a sentence structure like > Oh cool! That means that llama-swap is now superseded/no longer needed? That shows that you've read and understand the message, gives you the double-check and might on top spark a conversation about how these solutions compare. Plus that if the guy you're commenting too has spoken nonsense, they need to backpedal.
- emulio 2mo ago

deleted 2mo ago

[deleted]

toyg 2mo ago

> any api client can just select a model and the system does the right thing My machine can only load one model at a time. The loading/unloading times just don't seem worth the switch. I tend to use qwen3.6 for anything and that's it. Then again, I am a simple coder.