17 ms·

Sounds similar to vLLM: intermediate level of precision, tensor parallelism. vLLM is the most efficient server for concurrent mode. I don't know if TensorRT-LLM

Sounds similar to vLLM: intermediate level of precision, tensor parallelism. vLLM is the most efficient server for concurrent mode. I don't know if TensorRT-LLM also implements the paged kv cache. That makes interacting repeatedly with a reference text (and long chats) very efficient. I got up to 3000 T/s prefix and 400 T/s completion on 30 concurrent requests running on 2xA6000. So it is good for batching mode or agents that run many requests.

kkielhofner 3y ago
vLLM has healthy competition. Not affiliated but try lmdeploy: https://github.com/InternLM/lmdeploy https://github.com/InternLM/lmdeploy In my testing it’s significantly faster and more memory efficient than vLLM when configured with AWQ int4 and int8 KV cache. If you look at the PRs, issues, etc you’ll see there are many more optimizations in the works. That said there are also PRs and issues for some of the lmdeploy tricks in vllm as well (AWQ, Triton Inference Server, etc). I’m really excited to see where these projects go!
- KaoruAoiShiho 3y ago
  Does TensorRT-LLM improvements stack on top of lmdeploy or is it a replacement/competitor? Cause if it's the latter it's not so exciting since lmdeploy already advertises huge speedups.
  - kkielhofner 3y ago
    lmdeploy and vllm have custom backends for Nvidia Triton Inference Server, which then actually serves up models. lmdeploy is a little more mature as it essentially uses Triton by default but I expect vllm to come along quickly as Triton Inference Server has been the "go to" for high scale and high performance serving of models for years for a variety of reasons. It appears that we'll see TensorRT-LLM integrated in Triton Inference Server directly: https://github.com/vllm-project/vllm/issues/541 https://github.com/vllm-project/vllm/issues/541 How all of this will interact (vllm/lmdeploy backends + Triton + TensorRT-LLM) isn't quite clear to me right now.

kkielhofner 3y ago

vLLM has healthy competition. Not affiliated but try lmdeploy: https://github.com/InternLM/lmdeploy https://github.com/InternLM/lmdeploy In my testing it’s significantly faster and more memory efficient than vLLM when configured with AWQ int4 and int8 KV cache. If you look at the PRs, issues, etc you’ll see there are many more optimizations in the works. That said there are also PRs and issues for some of the lmdeploy tricks in vllm as well (AWQ, Triton Inference Server, etc). I’m really excited to see where these projects go!

KaoruAoiShiho 3y ago
Does TensorRT-LLM improvements stack on top of lmdeploy or is it a replacement/competitor? Cause if it's the latter it's not so exciting since lmdeploy already advertises huge speedups.
- kkielhofner 3y ago
  lmdeploy and vllm have custom backends for Nvidia Triton Inference Server, which then actually serves up models. lmdeploy is a little more mature as it essentially uses Triton by default but I expect vllm to come along quickly as Triton Inference Server has been the "go to" for high scale and high performance serving of models for years for a variety of reasons. It appears that we'll see TensorRT-LLM integrated in Triton Inference Server directly: https://github.com/vllm-project/vllm/issues/541 https://github.com/vllm-project/vllm/issues/541 How all of this will interact (vllm/lmdeploy backends + Triton + TensorRT-LLM) isn't quite clear to me right now.