4 ms·
TensorRT-LLM runtime now open-source
- mmoskal 2y agoPreviously, the "Executor" runtime was shipped as binary blobs. This is the bit that schedules requests and manages KV cache (similar to vLLM or SGLang server).
- deleted 2y ago[deleted]
loading thread…