3 ms·
JetSpec Enables Up to 9.64x Lossless LLM Inference Speedup with Up to 1000TPS
- snyhlxde 3mo agoWe find speculative decoding can push LLM generation latency to extreme by co-optimizing drafting cost and drafting quality with causal parallel tree drafting. JetSpec reaches up to 9.64× end-to-end speedup on MATH-500 and 4.58× on open-ended chat while keeping lossless. With CUDA graph and kernel optimizations, JetSpec further translates to around 1000 TPS on a single B200 GPU.