4 ms·
I don't understand why all of their hubub is about tokens per second (which relies on a lot more than just hardware) instead of FLOPs/memory bandwidth/etc.
by zaptrem 2y ago
I don't understand why all of their hubub is about tokens per second (which relies on a lot more than just hardware) instead of FLOPs/memory bandwidth/etc.
- consumer451 2y agoThey do make their own hardware, but of note is that they stopped selling hardware and instead pivoted to only selling API access, 4 months ago. Hence no need to describe anything aside from tokens/sec? https://news.ycombinator.com/item?id=39964590 https://news.ycombinator.com/item?id=39964590
- mrtksn 2y agoMaybe because that's how its measured in the LLM community? It's all about tokens per second.
- elorant 2y agoThey provide those numbers too. Memory bandwidth is at whooping 80 TBs/second.