8 ms·
Sorry, I'm a bit naïve about all of this. Why is this impressive? Can this result not be achieved by throwing more compute at the problem to speed up responses
by SeanAnderson 3y ago
Sorry, I'm a bit naïve about all of this.
Why is this impressive? Can this result not be achieved by throwing more compute at the problem to speed up responses? Isn't the fact that there is a queue when under load just indicative that there's a trade-off between "# of request to process per unit of time" and "amount of compute to put into a response to respond quicker"?
https://raw.githubusercontent.com/NVIDIA/TensorRT-LLM/rel/docs/source/blogs/media/TRT_LLM_v0-5-0_H100vA100_tps.png https://raw.githubusercontent.com/NVIDIA/TensorRT-LLM/rel/do...
This chart from NVIDIA implies their H100 runs llama v2 70B at >500 tok/s.
- moffkalast 3y agoI think NVidia is listing max throughput in terms of batching, so e.g. 50 tok/s for 10 different prompts at the same time. Groq LPUs definitely outerform an H100 in raw speed. But fundamentally it's a system that only has 10x the speed for 500x the price, made by a company that runs a blockchain and is trying to heavily market what were intended to be crypto mining chips for LLM inference. It's really quite a funny coincidence that when someone amazed posts this weekly link there's an army of Groq engineers at the ready in the comments ready to say everything and anything.
- tome 3y agoGroq does not run a blockchain and our chips were never intended for crypto mining.
- moffkalast 3y agohttps://www.livecoinwatch.com/price/GroqAI-GROQ https://www.livecoinwatch.com/price/GroqAI-GROQ I suppose that's someone else then? If that's true, then with this and Elon's Grok it's surprising the US Patent office hasn't taken your trademark away yet for not adequately defending it from infringement.
- deleted 3y ago[deleted]
- MasterScrat 3y agoScaling up compute can improve throughput, but can't easily improve latency between tokens. Generation is usually bottlenecked by the time it takes to go through the network for each token. To speed that up, you need to perform these computations faster, which is a hard problem after you've exhausted all the obvious options (use the fastest accelerator you can find, cache what you can etc).
- SeanAnderson 3y agoYeah. That makes sense, thank you for clarifying. I updated my original post with a chart from NVIDIA which highlights the H100's capabilities. It doesn't seem unreasonable to expect a 7B model to run at 500 tok/s on that hardware.
- snowfield 3y agoThis is a 50B model. (Mixtral 8x7b)
- SeanAnderson 3y agoOh, sorry, I assumed the 8 was for quantization. 8x7b is a new syntax for me. Still, the NVIDIA chart shows Llama v2 70B at 750 tok/s, no?
- tome 3y agoI guess that's total throughput, rather than per user? You can increase total throughput by scaling horizontally. You can't increase throughput per user that way.
- qeternity 3y agoAt batch size 1 LLMs are memory bandwidth bound, not compute bound…as in you spend most time waiting for model weights to load from vram. At higher batch sizes this flips. But this is why Groq is built around large numbers of chips with small amount of very fast sram.
- tome 3y agoLLM inference is inherently a sequential problem. You can't speed it up by doing more in parallel. You can't generate the 101st token before you've generated the 100th.
- NorwegianDude 3y agoTechnically, I guess you can use speculative execution to speed it up, and in that way take a guess at what the 100th token will be and start on the 101st token at the same time? Though it probably has it's own unforeseen challenges. Everything is predictable with enough guesses.
- _diq5 3y agoPeople are pretty cagey about what they use in production, but yes, speculative sampling can offer massive speedups in inference
- Aeolun 3y agoThey’re using several hundred cards here. Clearly there is ‘something’ that can be done in parallel.
- SushiHippie 3y agoI guess it depends on how much the infrastracture from TFA costs, as the H100 only costs ~$3300 to produce, but gets sold for ~$30k on average. https://www.hpcwire.com/2023/08/17/nvidia-h100-are-550000-gpus-enough-for-this-year/ https://www.hpcwire.com/2023/08/17/nvidia-h100-are-550000-gp...
- nabakin 3y agoThere's a difference between token throughput and latency. Token throughput is the token throughput of the whole GPU/system and latency is the token throughput for an individual user. Groq offers extremely low latency (aka extremely high token throughput per user) but we still don't have numbers on the token throughput of their entire system. Nvidia's metrics here on the other hand, show us the token throughput of the whole GPU/system. So, in reality, while you might be able to get 1.5k t/s on an H100, the latency (token throughput per user) will be something much lower like 20 t/s. The really important metric to look for is cost per token because even though Groq is able to run at low latency, that doesn't mean it's able to do it cheaply. Determining the cost per token can be done many ways but a useful way for us is approximately the cost of the system divided by the total token throughput of the system per second. We don't have the total token throughput per second of Groq's system so we can't really say how efficient it is. It could very well be that Groq is subsidizing the cost of their system to lower prices and gain PR and will increase their prices later on.
- frozenport 3y agohttps://wow.groq.com/artificialanalysis-ai-llm-benchmark-doubles-axis-to-fit-new-groq-lpu-inference-engine-performance-results/ https://wow.groq.com/artificialanalysis-ai-llm-benchmark-dou... Seems to have it. Looks cost competitive but a lot faster.
- nabakin 3y agoPeople are using throughput and latency differently in different locations/contexts. Here they are referring to token throughput per user and first token/chunk latency. They don't mention the token throughput of the entire 576-chip system[0] that runs Llama 2 70b which would be the number we're looking for. [0] https://news.ycombinator.com/item?id=38742581 https://news.ycombinator.com/item?id=38742581