3 ms·
The ratio of DRAM bandwidth to compute is constant no matter how many A100s you use because each A100 comes with its own dedicated DRAM. As a result GPT-3 would
by modeless 6y ago
The ratio of DRAM bandwidth to compute is constant no matter how many A100s you use because each A100 comes with its own dedicated DRAM. As a result GPT-3 would likely be faster to train on an A100 cluster with enough DRAM to hold the parameters. But Resnet-50 and WaveNet would likely be much faster to train on Cerebras because they fit in the SRAM. In fact, I believe even the biggest state-of-the-art Imagenet classification models would fit.