3 ms·

Sorry I meant Groq custom hardware, not Grok! I don't see any latency comparisons in the link

• danpalmer 11mo ago
  The link is just to the book, the details are scattered throughout. That said the page on GPUs specifically speaks to some of the hardware differences and how TPUs are more efficient for inference, and some of the differences that would lead to lower latency. https://jax-ml.github.io/scaling-book/gpus/#gpus-vs-tpus-at-the-chip-level https://jax-ml.github.io/scaling-book/gpus/#gpus-vs-tpus-at-... Re: Groq, that's a good point, I had forgotten about them. You're right they too are doing a TPU-style systolic array processor for lower latency.