4 ms·
You might be conflating throughput with latency. 100k tok/s is very different to 1 tok/10us.
by taneq 4mo ago
You might be conflating throughput with latency. 100k tok/s is very different to 1 tok/10us.
- ssivark 4mo agoWhen doing auto regressive inference, how often do you do a CUDA kernel call? What is the main bottleneck at the throughputs you're operating?