3 ms·
Last I looked vLLM didn't work on a Mac.
by simonw 2y ago
Last I looked vLLM didn't work on a Mac.
- mitjam 2y agoAfaik vllm is for concurrent serving with batched inference for higher throughput, not single-user inference. I doubt inference throughput is higher with single prompts at a time than Ollama. Update: this is a good Intro to continuous batching in llm inference: https://www.anyscale.com/blog/continuous-batching-llm-inference https://www.anyscale.com/blog/continuous-batching-llm-infere...
- Der_Einzige 2y agoIt is much faster on single prompts than ollama. 3X is not unheard of