3 ms·
The significant convenience benefits outweigh the higher TPS that vLLM offers in the context of my single machine homelab GPU server. If I was hosting it for so
by Zambyte 2y ago
The significant convenience benefits outweigh the higher TPS that vLLM offers in the context of my single machine homelab GPU server. If I was hosting it for something more critical than just myself and a few friends chatting with it, sure. Being able to just paste a model name into Open WebUI and run it is important to me though.
It is important to know about both to decide between the two for your use case though.
- Der_Einzige 2y agoRunning any HF model on vllm is as simple as pasting a model name into one command in your terminal.
- Zambyte 2y agoWhat command is it? Because that was not at all my experience.
- Der_Einzige 2y agoVllm serve… huggingface gives run instructions for every model with vllm on their website.
- Zambyte 2y agoHow do I serve multiple models? I can pick from dozens of models that I have downloaded through Open WebUI.
- iAMkenough 2y agoHad to build it from source to run on my Mac, and the experimental support doesn't seem to include these latest Gemma 3 QAT models on Apple Silicon.