3 ms·
>There are enough transistors, and enough RAM on the wafer to fit the entire model. Not the entire 70b fp16 model. It'd take 148GB of RAM to hold the entire mo
by aurareturn 2y ago
>There are enough transistors, and enough RAM on the wafer to fit the entire model.
Not the entire 70b fp16 model. It'd take 148GB of RAM to hold the entire model. Each Cerebras wafer chip has 44GB of SRAM. You need 4 of them chained together to hold the entire model.
- aurareturn 2y agoCorrection: 40GB of SRAM each. https://www.anandtech.com/show/16626/cerebras-unveils-wafer-scale-engine-two-wse2-26-trillion-transistors-100-yield https://www.anandtech.com/show/16626/cerebras-unveils-wafer-...
- jrnx 2y ago40 GB was the CS-2. CS-3 has 44 GB https://cerebras.ai/blog/cerebras-cs3 https://cerebras.ai/blog/cerebras-cs3