3 ms·
Friends Don't Let Friends Use Ollama https://news.ycombinator.com/item?id=47788385 https://news.ycombinator.com/item?id=47788385
by cube00 2mo ago
Friends Don't Let Friends Use Ollama https://news.ycombinator.com/item?id=47788385 https://news.ycombinator.com/item?id=47788385
- lenerdenator 2mo agoWhat do you use instead?
- reilly3000 2mo agotry oMLX or vMLX - both great projects that offer some amazing performance optimizations for Apple Silicon that utilize UMA and NVME caching efficiently. https://omlx.ai https://omlx.ai https://vmlx.net https://vmlx.net That said, it's been a few weeks since I've looked so maybe llama.cpp has those features now... they really do move that quickly.
- computershit 2mo agoI use llama.cpp w/ llama-swap https://github.com/ggml-org/llama.cpp https://github.com/ggml-org/llama.cpp https://github.com/mostlygeek/llama-swap https://github.com/mostlygeek/llama-swap
- LeBit 2mo agoThanks for the link to llama-swap. Didn’t know about it and will definitely install it.
- rancor 2mo agoFYI, llama-server can now be run in router mode so llama-swap is probably only needed for more exotic scenarios.
- kingo55 2mo agoI'm running it in router mode, but people on Reddit were recommending people use llama-swap instead. Am I missing something by using router mode?
- mudkipdev 2mo agoI believe it's useful for running multiple llama.cpp forks at the same time (e.g. a model you want requires special patching)
- computershit 2mo agoI’m using llama-swap because it can manage arbitrary backends, not just llama-server instances. I have llama.cpp chat and embedding models running alongside whisper-server all behind a single endpoint with per-model TTLs so they don't fight over the limited vram I have available on this box. Native routing could replace the llama.cpp part but not whisper so I guess I'm exotic ;)
- soupspaces 2mo agoThe link in that discussion has a section about that
- nickthegreek 2mo agolm studio
- SwellJoe 2mo agoIf you need a GUI, Unsloth Studio and LM Studio are both great frontends for llama.cpp. If you don't need a GUI, llama.cpp is the business for single-user deployments. Easy to use, always gets new model support very quickly, built-in Hugging Face client/cache support, works on probably everything (Mac, ROCm, CUDA, Vulkan, etc.).
- scotty79 2mo agoJan is so much better.
- LeBit 2mo agoWhen that article was first published , I started looking into llama.cpp. With the help of an LLM I researched the knobs to turn that made most sense. Things went from "local models are useless unless you have a 512GB GPU I guess" to "oh.. I can do a lot of stuff locally now!"
- chorizo 2mo agoHad a similar experience. Llama.cpp compiled natively; parameter sweep to find best options fitting my use case for the qwen models with 16GB VRAM. The whole thing packaged into a portable container.
- mark_l_watson 2mo agoFrom the Ollama docs for this new model: Ollama's MLX engine provides state-of-the-art performance on Apple Silicon, with support for DFlash and image input: ollama run muse-glimmer:30b-mlx