4 ms·
This is very interesting and yet not at the same time. This looks to be optimized for single-stream LLM traffic which is not viable to serve in a production set
by beffjezos 3mo ago
This is very interesting and yet not at the same time. This looks to be optimized for single-stream LLM traffic which is not viable to serve in a production setting. It's only interesting to hobbyists that want to run the model locally.
It's genuinely neat that AI can find the right optimization pathways in an AMD inference server to unlock this but at the same token (pun-intended) this is a classic case of benchmark hacking that doesn't stand up to real-world application.
- technoabsurdist 3mo agohi yes it’s not optimized for single stream it’s optimized for total node throughput
- beffjezos 3mo agoOh, that's much better then. A good metric to share is the tokens per second per user for the node rather than the total throughput of the node. It disambiguates what's being optimized for much better than your blog post currently does.
- technoabsurdist 3mo agosounds good feedback taken, thanks beffjezos
- foobar10000 3mo agoWell, for a lot of agentic stuff nowadays, having 250k-500K context is where things live - and the benchmarks don't really show that unfortunately - but they could :)
- wmf 3mo agoYou got it backwards; it's ~200 on single stream so the 2,600 is achieved with ~13 streams.
- beffjezos 3mo agoYeah that makes sense. I'm more familiar with seeing tok/s/user + TTFT rather than the total node throughput.