3 ms·
vLLM has healthy competition. Not affiliated but try lmdeploy: https://github.com/InternLM/lmdeploy https://github.com/InternLM/lmdeploy In my testing it’s si
by kkielhofner 3y ago
vLLM has healthy competition. Not affiliated but try lmdeploy:
https://github.com/InternLM/lmdeploy https://github.com/InternLM/lmdeploy
In my testing it’s significantly faster and more memory efficient than vLLM when configured with AWQ int4 and int8 KV cache.
If you look at the PRs, issues, etc you’ll see there are many more optimizations in the works. That said there are also PRs and issues for some of the lmdeploy tricks in vllm as well (AWQ, Triton Inference Server, etc).
I’m really excited to see where these projects go!
- KaoruAoiShiho 3y agoDoes TensorRT-LLM improvements stack on top of lmdeploy or is it a replacement/competitor? Cause if it's the latter it's not so exciting since lmdeploy already advertises huge speedups.
- kkielhofner 3y agolmdeploy and vllm have custom backends for Nvidia Triton Inference Server, which then actually serves up models. lmdeploy is a little more mature as it essentially uses Triton by default but I expect vllm to come along quickly as Triton Inference Server has been the "go to" for high scale and high performance serving of models for years for a variety of reasons. It appears that we'll see TensorRT-LLM integrated in Triton Inference Server directly: https://github.com/vllm-project/vllm/issues/541 https://github.com/vllm-project/vllm/issues/541 How all of this will interact (vllm/lmdeploy backends + Triton + TensorRT-LLM) isn't quite clear to me right now.