4 ms·
Thanks, thats awesome. Would you say vLLM on one GPU is as fast or faster than llama.cpp? have you tried mlc-llm?
by freediver 3y ago
Thanks, thats awesome. Would you say vLLM on one GPU is as fast or faster than llama.cpp? have you tried mlc-llm?
- Tostino 3y agoLlama.cpp has faster single thread inference, but I actually haven't tried their batched inference yet, so I don't know what throughput I can get through it.