2 ms·
We leverage Flash Decoding (https://crfm.stanford.edu/2023/10/12/flashdecoding.html https://crfm.stanford.edu/2023/10/12/flashdecoding.html) in TensorRT-LLM to
by rushingcreek 3y ago
We leverage Flash Decoding (https://crfm.stanford.edu/2023/10/12/flashdecoding.html https://crfm.stanford.edu/2023/10/12/flashdecoding.html) in TensorRT-LLM to achieve 100 tokens per second on H100s.