3 ms·

You can say the same about a regular GPU, no?

• modeless 6y ago
  Sure, and memory bandwidth can be a bottleneck for GPUs. But it will be a 100x worse bottleneck for Cerebras because the ratio of compute to DRAM bandwidth is that much higher.
  • p1esk 6y ago
    If you compare CS-2 to a workstation with a single A100 in it, sure. But we should compare CS-2 to something that provides the same amount of compute, right? What's the ratio of compute to bandwidth in a rack full of A100 chips? What are the bottlenecks in that rack: DRAM bandwidth? I/O? Internode communication? If we want to train GPT-3 (or WaveNet, or Resnet-50), what's more efficient: a rack of A100s or a single CS-2 box? Which one is faster, assuming the same cost and power consumption? Right now I'd probably pick the A100 rack as the winner, but long term I'm interested in what Cerebras is doing. If you think about it, our brain is more like a wafer scale chip than a rack of GPU cards.
    • modeless 6y ago
      The ratio of DRAM bandwidth to compute is constant no matter how many A100s you use because each A100 comes with its own dedicated DRAM. As a result GPT-3 would likely be faster to train on an A100 cluster with enough DRAM to hold the parameters. But Resnet-50 and WaveNet would likely be much faster to train on Cerebras because they fit in the SRAM. In fact, I believe even the biggest state-of-the-art Imagenet classification models would fit.