2 ms·
llama.cpp has router mode these days, swapping out models for you, you no longer need llama-swap.
by NorwegianDude 2mo ago
llama.cpp has router mode these days, swapping out models for you, you no longer need llama-swap.
- dangoljames 2mo agoI use it on the daily. Haven't exactly missed ollama.
- cptskippy 2mo agoThe router mode and matrix routing in llama.cpp is still early days and it can't easily juggle multiple models as easily as llama-swap so there's still benefits if you're using 24-32GB cards that can run multiple models simultaneously. The gulf is quickly shrinking though and it seems like the need for llama-swap will disappear soon.