3 ms·
If you can get away with it, you really should stick to llama-server alone. When they added support the ability to load multiple models + an idle timeout to unl
by tredre3 2mo ago
If you can get away with it, you really should stick to llama-server alone. When they added support the ability to load multiple models + an idle timeout to unload them, it finally did it for me.
Adding llama-swap adds a fair amount of complexity that is unpleasant to debug and its documentation isn't very good (the only real doc is the sample config file).
- cptskippy 2mo agoThe native router matrix for llama.cpp is still pretty immature (though evolving quickly) and I plan to switch once it reaches feature parity. Personally, I don't see the value in idle timeouts or time based eviction. It doesn't improve model load performance but does mean you're more likely to incur a load penalty for any request. I generally use the same two models for my agentic workloads and they sit comfortably next to one another in the 32GB of VRAM on my GPU. If I need to load a larger model, llama-swap can easily eject both models if necessary.