3 ms·

Groq devices are really well set up for small-batch-size inference because of the use of SRAM. I'm not so convinced they have a Tok/sec/$ advantage at all, tho

Groq devices are really well set up for small-batch-size inference because of the use of SRAM. I'm not so convinced they have a Tok/sec/$ advantage at all, though, and especially at medium to large batch sizes which would be the groups who can afford to buy so much silicon. I assume given the architecture that Groq actually doesn't get any faster for batch sizes >1, and Nvidia cards do get meaningfully higher throughput as batch size gets into the 100's.

nabakin 3y ago
I've been thinking the same but on the other hand, that would mean they are operating at a huge loss which doesn't scale
foundval 3y ago
(Groq Employee) It's hard to discuss Tok/sec/$ outside of the context of a hardware sales engagement. This is because the relationship between Tok/s/u, Tok/s/system, Batching, and Pipelining is a complex one that involves compute utilization, network utilization, and (in particular) a host of compilation techniques that we wouldn't want to share publicly. Maybe we'll get to that level of transparency at some point, though! As far as Batching goes, you should consider that with synchronous systems, if all the stars align, Batch=1 is all you need. Of course, the devil is in the details, and sometimes small batch numbers still give you benefits. But Batch 100's generally gives no advantages. In fact, the entire point of developing deterministic hardware and synchronous systems is to avoid batching in the first place.
frozenport 3y ago
I assume given the architecture that Groq actually doesn't get any faster for batch sizes >1 I guess if you don't have any extra junk you can pack more processing into the chip?
- foundval 3y ago
  (Groq Employee) Yes! Determinism + Simplicity are superpowers for ALU and interconnect utilization rates. This system is powered by 14nm chips, and even the interconnects aren't best in class. We're just that much better at squeezing tokens out of transistors and optic cables than GPUs are - and you can imagine the implications on Watt/Token. Anyways.. wait until you see our 4nm. :)

nabakin 3y ago

I've been thinking the same but on the other hand, that would mean they are operating at a huge loss which doesn't scale

foundval 3y ago

(Groq Employee) It's hard to discuss Tok/sec/$ outside of the context of a hardware sales engagement. This is because the relationship between Tok/s/u, Tok/s/system, Batching, and Pipelining is a complex one that involves compute utilization, network utilization, and (in particular) a host of compilation techniques that we wouldn't want to share publicly. Maybe we'll get to that level of transparency at some point, though! As far as Batching goes, you should consider that with synchronous systems, if all the stars align, Batch=1 is all you need. Of course, the devil is in the details, and sometimes small batch numbers still give you benefits. But Batch 100's generally gives no advantages. In fact, the entire point of developing deterministic hardware and synchronous systems is to avoid batching in the first place.

frozenport 3y ago

I assume given the architecture that Groq actually doesn't get any faster for batch sizes >1 I guess if you don't have any extra junk you can pack more processing into the chip?

foundval 3y ago
(Groq Employee) Yes! Determinism + Simplicity are superpowers for ALU and interconnect utilization rates. This system is powered by 14nm chips, and even the interconnects aren't best in class. We're just that much better at squeezing tokens out of transistors and optic cables than GPUs are - and you can imagine the implications on Watt/Token. Anyways.. wait until you see our 4nm. :)