2 ms·

When aiming for 100k tok/s, you would still have CUDA overheads (on the order of microseconds) -- which might become the bottleneck, even if you do everything e

by ssivark 4mo ago

When aiming for 100k tok/s, you would still have CUDA overheads (on the order of microseconds) -- which might become the bottleneck, even if you do everything else right with the inference architecture. How are you planning to overcome that? EDIT: Oh, on second read, do you mean you're running the model on an FPGA?

taneq 4mo ago
You might be conflating throughput with latency. 100k tok/s is very different to 1 tok/10us.
- ssivark 4mo ago
  When doing auto regressive inference, how often do you do a CUDA kernel call? What is the main bottleneck at the throughputs you're operating?