7 ms·
AFAIK they are a hardware company producing special chips for processing AI tasks. Their response time is incredibly fast, like 500t-900t/s for LLMs. They're no
by mrtksn 2y ago
AFAIK they are a hardware company producing special chips for processing AI tasks. Their response time is incredibly fast, like 500t-900t/s for LLMs. They're not just a company that buys in Nvidia cards for serving as a backend.
In other words, they are competing with Nvidia, not the million startups using Nvidia.
- deleted 2y ago[deleted]
- zaptrem 2y agoI don't understand why all of their hubub is about tokens per second (which relies on a lot more than just hardware) instead of FLOPs/memory bandwidth/etc.
- consumer451 2y agoThey do make their own hardware, but of note is that they stopped selling hardware and instead pivoted to only selling API access, 4 months ago. Hence no need to describe anything aside from tokens/sec? https://news.ycombinator.com/item?id=39964590 https://news.ycombinator.com/item?id=39964590
- mrtksn 2y agoMaybe because that's how its measured in the LLM community? It's all about tokens per second.
- elorant 2y agoThey provide those numbers too. Memory bandwidth is at whooping 80 TBs/second.