3 ms·
Running any HF model on vllm is as simple as pasting a model name into one command in your terminal.
by Der_Einzige 2y ago
Running any HF model on vllm is as simple as pasting a model name into one command in your terminal.
- Zambyte 2y agoWhat command is it? Because that was not at all my experience.
- Der_Einzige 2y agoVllm serve… huggingface gives run instructions for every model with vllm on their website.
- Zambyte 2y agoHow do I serve multiple models? I can pick from dozens of models that I have downloaded through Open WebUI.
- iAMkenough 2y agoHad to build it from source to run on my Mac, and the experimental support doesn't seem to include these latest Gemma 3 QAT models on Apple Silicon.