4 ms·
the only reason they are fast is because the models they host are severely quantized so i've heard.
by sergiotapia 1y ago
the only reason they are fast is because the models they host are severely quantized so i've heard.
- jacob019 1y agoHuh. I heard a podcast with the founder talking about their custom hardware, but quantization would explain it.
- christianqchung 1y agoQuantization alone does not explain it. It's mostly custom hardware[0]. [0] https://groq.com/the-groq-lpu-explained/ https://groq.com/the-groq-lpu-explained/
- behnamoh 1y agothey responded to my tweet last year and said they didn't quantize the models.
- boroboro4 1y agoIt's very hard to find right now but I'm sure they said they don't quantize KV cache, but their weights are in fp8.
- zargon 1y agoWhy repeat this nonsense when it’s so trivial to just check. The reason Groq is fast is because they employ absolutely ludicrous amounts of SRAM. (Which is 10 times faster than the fastest VRAM.)