5 ms·
Available on ollama: https://ollama.com/library/gemma3 https://ollama.com/library/gemma3
by emrah 1y ago
Available on ollama: https://ollama.com/library/gemma3 https://ollama.com/library/gemma3
- jinay 1y agoMake sure you're using the "-it-qat" suffixed models like "gemma3:27b-it-qat"
- Zambyte 1y agoHere are the direct links: https://ollama.com/library/gemma3:27b-it-qat https://ollama.com/library/gemma3:27b-it-qat https://ollama.com/library/gemma3:12b-it-qat https://ollama.com/library/gemma3:12b-it-qat https://ollama.com/library/gemma3:4b-it-qat https://ollama.com/library/gemma3:4b-it-qat https://ollama.com/library/gemma3:1b-it-qat https://ollama.com/library/gemma3:1b-it-qat
- ein0p 1y agoThanks. I was wondering why my open-webui said that I already had the model. I bet a lot of people are making the same mistake I did and downloading just the old, post-quantized 27B.
- Der_Einzige 1y agoHow many times do I have to say this? Ollama, llamacpp, and many other projects are slower than vLLM/sglang. vLLM is a much superior inference engine and is fully supported by the only LLM frontends that matter (sillytavern). The community getting obsessed with Ollama has done huge damage to the field, as it's ineffecient compared to vLLM. Many people can get far more tok/s than they think they could if only they knew the right tools.
- janderson215 1y agoI did not know this, so thank you. I read a blogpost a while back that encouraged using Ollama and never mention vLLM. Do you recommend reading any particular resource?
- m00dy 1y agoOllama is definitely not for production loads but vLLm is.
- Zambyte 1y agoThe significant convenience benefits outweigh the higher TPS that vLLM offers in the context of my single machine homelab GPU server. If I was hosting it for something more critical than just myself and a few friends chatting with it, sure. Being able to just paste a model name into Open WebUI and run it is important to me though. It is important to know about both to decide between the two for your use case though.
- Der_Einzige 1y agoRunning any HF model on vllm is as simple as pasting a model name into one command in your terminal.
- Zambyte 1y agoWhat command is it? Because that was not at all my experience.
- Der_Einzige 1y agoVllm serve… huggingface gives run instructions for every model with vllm on their website.
- Zambyte 1y agoHow do I serve multiple models? I can pick from dozens of models that I have downloaded through Open WebUI.
- iAMkenough 1y agoHad to build it from source to run on my Mac, and the experimental support doesn't seem to include these latest Gemma 3 QAT models on Apple Silicon.
- oezi 1y agoWhy is sillytavern the only LLM frontend which matters?
- GordonS 1y agoI tried sillytavern a few weeks ago... wow, that is an "interesting" UI! I blundered around for a while, couldn't figure out how to do anything useful... and then installed LM Studio instead.
- imtringued 1y agoI personally thought the lorebook feature was quite neat and then quickly gave up on it because I couldn't get it to trigger, ever. Whatever those keyword things are, they certainly don't seem to be doing any form of RAG.
- Der_Einzige 1y agoIt supports more sampler and other settings than anyone else.
- ach9l 1y agoinstead of ranting, maybe explain how to make a qat q4 work with images in vllm, afaik it is not yet possible
- oezi 1y agoSomebody in this thread mentioned 20.x tok/s on ollama. What are you seeing in vLLM?
- Zambyte 1y agoFWIW I'm getting 29 TPS on Ollama on my 7900 XTX with the 27b qat. You can't really compare inference engine to inference engine without keeping the hardware and model fixed. Unfortunately Ollama and vLLM are therefore incomparable at the moment, because vLLM does not support these models yet. https://github.com/vllm-project/vllm/issues/16856 https://github.com/vllm-project/vllm/issues/16856
- simonw 1y agoLast I looked vLLM didn't work on a Mac.
- mitjam 1y agoAfaik vllm is for concurrent serving with batched inference for higher throughput, not single-user inference. I doubt inference throughput is higher with single prompts at a time than Ollama. Update: this is a good Intro to continuous batching in llm inference: https://www.anyscale.com/blog/continuous-batching-llm-inference https://www.anyscale.com/blog/continuous-batching-llm-infere...
- Der_Einzige 1y agoIt is much faster on single prompts than ollama. 3X is not unheard of
- prometheon1 1y agoFrom the HN guidelines: https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html > Be kind. Don't be snarky. > Please don't post shallow dismissals, especially of other people's work. In my opinion, your comment is not in line with the guidelines. Especially the part about sillytavern being the only LLM frontend that matters. Telling the devs of any LLM frontend except sillytavern that their app doesn't matter seems exactly like a shallow dismissal of other people's work to me.
- mschoch 1y ago[dead]