4 ms·
The one thing I totally don't understand about this architecture is the meager 40GB of RAM. I understand it's SRAM and the high memory bandwidth allows for redu
by LeegleechN 5y ago
The one thing I totally don't understand about this architecture is the meager 40GB of RAM. I understand it's SRAM and the high memory bandwidth allows for reduced replication, but almost every workload I can imagine this addressing is still going to have a larger working set than that. I guess they still haven't figured out how to stack HBM on top of the wafer, and they're just carefully avoiding calling attention to this extreme (factor of 1000) weakness.
- modeless 5y agoYeah I think you couldn't train GPT-3 on this. You need something incredibly compute and memory bandwidth intensive, but with relatively few parameters. But there ought to be some interesting architectures to discover in that direction. If nothing else you could do crazy fast hyperparameter optimization on smaller architectures and get much better performance out of them. The architectures we have are shaped by the hardware they run on, I think more than people realize. Different hardware will need different architectures to be fully utilized. Unfortunately there are so few of these chips in existence that it will be hard to get enough people using them to really discover how to best use them.
- mark_l_watson 5y agoI agree, but consider this: there will hopefully be a new paradigm that uses fewer parameters but some mechanism like transfer learning to allow more modular training and composition of models. In an old job, I wrote many US patents and if I still worked there, then I would persue this idea. Now, I am just throwing this idea up in the air hoping it may be useful.
- p1esk 5y agoI couldn't find any info about it, but I imagine it does have plenty of system RAM next to the chip, what makes you think otherwise?
- modeless 5y agoThe bandwidth to RAM will be many orders of magnitude smaller than the bandwidth of on chip SRAM. You likely won't be able to use the full potential of this chip unless your problem can mostly fit in the SRAM.
- p1esk 5y agoYou can say the same about a regular GPU, no?
- modeless 5y agoSure, and memory bandwidth can be a bottleneck for GPUs. But it will be a 100x worse bottleneck for Cerebras because the ratio of compute to DRAM bandwidth is that much higher.
- p1esk 5y agoIf you compare CS-2 to a workstation with a single A100 in it, sure. But we should compare CS-2 to something that provides the same amount of compute, right? What's the ratio of compute to bandwidth in a rack full of A100 chips? What are the bottlenecks in that rack: DRAM bandwidth? I/O? Internode communication? If we want to train GPT-3 (or WaveNet, or Resnet-50), what's more efficient: a rack of A100s or a single CS-2 box? Which one is faster, assuming the same cost and power consumption? Right now I'd probably pick the A100 rack as the winner, but long term I'm interested in what Cerebras is doing. If you think about it, our brain is more like a wafer scale chip than a rack of GPU cards.
- modeless 5y agoThe ratio of DRAM bandwidth to compute is constant no matter how many A100s you use because each A100 comes with its own dedicated DRAM. As a result GPT-3 would likely be faster to train on an A100 cluster with enough DRAM to hold the parameters. But Resnet-50 and WaveNet would likely be much faster to train on Cerebras because they fit in the SRAM. In fact, I believe even the biggest state-of-the-art Imagenet classification models would fit.