4 ms·
How does the Groq PCIE Card work exactly? Does it use system ram to stream the model data to the card? How many T/s could one expect with e.g. 36000Mhz DDR4 Ram
by kkzz99 3y ago
How does the Groq PCIE Card work exactly? Does it use system ram to stream the model data to the card? How many T/s could one expect with e.g. 36000Mhz DDR4 Ram?
- tome 3y agoWe build out large systems where we stream in the model weights to the system once and then run multiple inferences on it. We don't really recommend streaming model weights repeatedly onto the chip because you'll lose the benefits of low latency.
- kkzz99 3y agoHow does that work when the card only has 230MB of SRAM?
- tome 3y agoWe connect hundreds of chips across several racks with fast interconnect.
- AhtiK 3y agoHow fast is the memory bandwidth of that fast interconnect?
- tome 3y agoHave a look at section 2.3 of our paper. Between any two chips we get 100 Gbps. The overall bandwidth depends on the connection topology used. I don't know if we make that public. https://wow.groq.com/wp-content/uploads/2023/05/GroqISCAPaper2022_ASoftwareDefinedTensorStreamingMultiprocessorForLargeScaleMachineLearning-1.pdf https://wow.groq.com/wp-content/uploads/2023/05/GroqISCAPape...