3 ms·
hi yes it’s not optimized for single stream it’s optimized for total node throughput
by technoabsurdist 3mo ago
hi yes it’s not optimized for single stream it’s optimized for total node throughput
- beffjezos 3mo agoOh, that's much better then. A good metric to share is the tokens per second per user for the node rather than the total throughput of the node. It disambiguates what's being optimized for much better than your blog post currently does.
- technoabsurdist 3mo agosounds good feedback taken, thanks beffjezos
- foobar10000 3mo agoWell, for a lot of agentic stuff nowadays, having 250k-500K context is where things live - and the benchmarks don't really show that unfortunately - but they could :)