2 ms·
We build out large systems where we stream in the model weights to the system once and then run multiple inferences on it. We don't really recommend streaming m
by tome 3y ago
We build out large systems where we stream in the model weights to the system once and then run multiple inferences on it. We don't really recommend streaming model weights repeatedly onto the chip because you'll lose the benefits of low latency.
- kkzz99 3y agoHow does that work when the card only has 230MB of SRAM?
- tome 3y agoWe connect hundreds of chips across several racks with fast interconnect.
- AhtiK 3y agoHow fast is the memory bandwidth of that fast interconnect?
- tome 3y agoHave a look at section 2.3 of our paper. Between any two chips we get 100 Gbps. The overall bandwidth depends on the connection topology used. I don't know if we make that public. https://wow.groq.com/wp-content/uploads/2023/05/GroqISCAPaper2022_ASoftwareDefinedTensorStreamingMultiprocessorForLargeScaleMachineLearning-1.pdf https://wow.groq.com/wp-content/uploads/2023/05/GroqISCAPape...