3 ms·
The wrapping of TensorRT-LLM alone is significant. I’ve been working with it for a while and it’s… Rough. That said it is extremely fast. With TensorRT-LLM an
by kkielhofner 3y ago
The wrapping of TensorRT-LLM alone is significant.
I’ve been working with it for a while and it’s… Rough.
That said it is extremely fast. With TensorRT-LLM and Triton Inference Server with conservation performance settings I get roughly 175 tokens/s on an RTX 4090 with Mistral-Instruct 7B. Following commits, PRs, etc I expect this to increase significantly in the future.
I’m actually working on a project to better package Triton and TensorRT-LLM and make it “name and model and press enter” level usable with support for embeddings models, Whisper, etc.