3 ms·
Peak H100 power consumption is 700W. Average power consumption of the groq card (from their own website) is 240W. With 576 chips it just doesn’t look good. How
by mlazos 3y ago
Peak H100 power consumption is 700W. Average power consumption of the groq card (from their own website) is 240W. With 576 chips it just doesn’t look good. How much is that millisecond perf gain worth it to end users?
That said I think their arch is super interesting. I just think that demo was way too hype when the actual system is pretty impractical.
- ein0p 3y agoSo? They aren't performing the same computation. You can't compare the two. What you can compare is power draw at an equivalent tokens/sec on the same model for the entire system. But you don't have that number.
- mlazos 3y agoI’m just estimating here from public numbers. My point is the power consumption could be 100W on that workload and the groq chip could be 1k, both ridiculously optimistic. The whole system is still crazy expensive. H100s will not have latency as fast, but in terms of concurrent users and TCO, I really don’t think groq will be worth it. You could probably get the same concurrent users and throughput with like 8 h100s. Latency won’t be as good, but price could be much lower.