3 ms·

https://wow.groq.com/artificialanalysis-ai-llm-benchmark-doubles-axis-to-fit-new-groq-lpu-inference-engine-performance-results/ https://wow.groq.com/artificiala

• nabakin 3y ago
  People are using throughput and latency differently in different locations/contexts. Here they are referring to token throughput per user and first token/chunk latency. They don't mention the token throughput of the entire 576-chip system[0] that runs Llama 2 70b which would be the number we're looking for. [0] https://news.ycombinator.com/item?id=38742581 https://news.ycombinator.com/item?id=38742581