3 ms·
Are you using 16bit for inference? How many tokens/second if you use 8bit? Given that SOTA models now use 4bit inference, can you do an estimation for 4bit + B
by aurareturn 10mo ago
Are you using 16bit for inference? How many tokens/second if you use 8bit?
Given that SOTA models now use 4bit inference, can you do an estimation for 4bit + Blackwell?
- mycelia 10mo agoHi! This benchmarking was done w/ DeepSeek-V3's published FP8 weights. And Blackwell performance is still being optimized. SGLang hit 14k/s/B200 though, pretty cool writeup here: https://lmsys.org/blog/2025-09-25-gb200-part-2/ https://lmsys.org/blog/2025-09-25-gb200-part-2/