4 ms·

There’s no comparison between those two though. You can go and get an RTX 3060 with $300 and run Llama3 8B. To do the same with Groq’s chips you’d need to buy 2

There’s no comparison between those two though. You can go and get an RTX 3060 with $300 and run Llama3 8B. To do the same with Groq’s chips you’d need to buy 25 of them at $20k each. To run something like the 70B model you need at least $6M in hardware.

recursivecaveat 2y ago
I don't think that's really a fair comparison. Like if you wanted to sell ball bearings you could spend $50 and start dripping lead into a bucket of water, or you could spend millions on a factory. The latter is definitely the better business model. What maters is quality of result (latency and tokens/s/user) and throughput per watt or capex dollar, not your minimum capex expense so long as you're running at any kind of scale.
- elorant 2y ago
  How is throughput per watt affordable? Each card has a TDP of 275 watts, and a maximum draw of 375. If you try to run a big model like Llama-3 400B you're looking at a cluster with 1,8k cards, and a total power draw of nearly 500KWs per hour.
eigenvalue 2y ago
Things don't have to be exactly in the same category to usefully compare them. Groq has emerged as the premiere way to host LLMs at scale at speeds that are dramatically faster than anyone else, which allows for all kinds of new and exciting applications that wouldn't work if you had to wait for 50tok/sec responses, but which feel magical at 500tok/sec. And although the exciting with LLMs seems to be around the training of them, I think if you look out a few years, vastly more FLOPs will be expended on inference than on training.
- lostmsu 2y ago
  Where can I find technical performance comparison? What models does Groq run at 500tok/sec? In what mode (e.g. batch size)? UPD. found the model, it is Mixtral 8x7B-32k. AFAIK 8xH100 will do 100+tok/sec with batch size 1. But that does not look too impressive for batch sizes higher than 1: https://www.baseten.co/blog/faster-mixtral-inference-with-tensorrt-llm-and-quantization/ https://www.baseten.co/blog/faster-mixtral-inference-with-te...

recursivecaveat 2y ago

I don't think that's really a fair comparison. Like if you wanted to sell ball bearings you could spend $50 and start dripping lead into a bucket of water, or you could spend millions on a factory. The latter is definitely the better business model. What maters is quality of result (latency and tokens/s/user) and throughput per watt or capex dollar, not your minimum capex expense so long as you're running at any kind of scale.

elorant 2y ago
How is throughput per watt affordable? Each card has a TDP of 275 watts, and a maximum draw of 375. If you try to run a big model like Llama-3 400B you're looking at a cluster with 1,8k cards, and a total power draw of nearly 500KWs per hour.

eigenvalue 2y ago

Things don't have to be exactly in the same category to usefully compare them. Groq has emerged as the premiere way to host LLMs at scale at speeds that are dramatically faster than anyone else, which allows for all kinds of new and exciting applications that wouldn't work if you had to wait for 50tok/sec responses, but which feel magical at 500tok/sec. And although the exciting with LLMs seems to be around the training of them, I think if you look out a few years, vastly more FLOPs will be expended on inference than on training.

lostmsu 2y ago
Where can I find technical performance comparison? What models does Groq run at 500tok/sec? In what mode (e.g. batch size)? UPD. found the model, it is Mixtral 8x7B-32k. AFAIK 8xH100 will do 100+tok/sec with batch size 1. But that does not look too impressive for batch sizes higher than 1: https://www.baseten.co/blog/faster-mixtral-inference-with-tensorrt-llm-and-quantization/ https://www.baseten.co/blog/faster-mixtral-inference-with-te...