4 ms·
How many times do I have to say this? Ollama, llamacpp, and many other projects are slower than vLLM/sglang. vLLM is a much superior inference engine and is ful
by Der_Einzige 2y ago
How many times do I have to say this? Ollama, llamacpp, and many other projects are slower than vLLM/sglang. vLLM is a much superior inference engine and is fully supported by the only LLM frontends that matter (sillytavern).
The community getting obsessed with Ollama has done huge damage to the field, as it's ineffecient compared to vLLM. Many people can get far more tok/s than they think they could if only they knew the right tools.
- janderson215 2y agoI did not know this, so thank you. I read a blogpost a while back that encouraged using Ollama and never mention vLLM. Do you recommend reading any particular resource?
- m00dy 2y agoOllama is definitely not for production loads but vLLm is.
- Zambyte 2y agoThe significant convenience benefits outweigh the higher TPS that vLLM offers in the context of my single machine homelab GPU server. If I was hosting it for something more critical than just myself and a few friends chatting with it, sure. Being able to just paste a model name into Open WebUI and run it is important to me though. It is important to know about both to decide between the two for your use case though.
- Der_Einzige 2y agoRunning any HF model on vllm is as simple as pasting a model name into one command in your terminal.
- Zambyte 2y agoWhat command is it? Because that was not at all my experience.
- Der_Einzige 2y agoVllm serve… huggingface gives run instructions for every model with vllm on their website.
- Zambyte 2y agoHow do I serve multiple models? I can pick from dozens of models that I have downloaded through Open WebUI.
- iAMkenough 2y agoHad to build it from source to run on my Mac, and the experimental support doesn't seem to include these latest Gemma 3 QAT models on Apple Silicon.
- oezi 2y agoWhy is sillytavern the only LLM frontend which matters?
- GordonS 2y agoI tried sillytavern a few weeks ago... wow, that is an "interesting" UI! I blundered around for a while, couldn't figure out how to do anything useful... and then installed LM Studio instead.
- imtringued 2y agoI personally thought the lorebook feature was quite neat and then quickly gave up on it because I couldn't get it to trigger, ever. Whatever those keyword things are, they certainly don't seem to be doing any form of RAG.
- Der_Einzige 2y agoIt supports more sampler and other settings than anyone else.
- ach9l 2y agoinstead of ranting, maybe explain how to make a qat q4 work with images in vllm, afaik it is not yet possible
- oezi 2y agoSomebody in this thread mentioned 20.x tok/s on ollama. What are you seeing in vLLM?
- Zambyte 2y agoFWIW I'm getting 29 TPS on Ollama on my 7900 XTX with the 27b qat. You can't really compare inference engine to inference engine without keeping the hardware and model fixed. Unfortunately Ollama and vLLM are therefore incomparable at the moment, because vLLM does not support these models yet. https://github.com/vllm-project/vllm/issues/16856 https://github.com/vllm-project/vllm/issues/16856
- simonw 2y agoLast I looked vLLM didn't work on a Mac.
- mitjam 2y agoAfaik vllm is for concurrent serving with batched inference for higher throughput, not single-user inference. I doubt inference throughput is higher with single prompts at a time than Ollama. Update: this is a good Intro to continuous batching in llm inference: https://www.anyscale.com/blog/continuous-batching-llm-inference https://www.anyscale.com/blog/continuous-batching-llm-infere...
- Der_Einzige 2y agoIt is much faster on single prompts than ollama. 3X is not unheard of
- prometheon1 2y agoFrom the HN guidelines: https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html > Be kind. Don't be snarky. > Please don't post shallow dismissals, especially of other people's work. In my opinion, your comment is not in line with the guidelines. Especially the part about sillytavern being the only LLM frontend that matters. Telling the devs of any LLM frontend except sillytavern that their app doesn't matter seems exactly like a shallow dismissal of other people's work to me.
- mschoch 2y ago[dead]