3 ms·

I'd love to see some metrics on whether this idea has any merit. Because obviously any problem that can be done on a massive wafer can be done on multiple small

I'd love to see some metrics on whether this idea has any merit. Because obviously any problem that can be done on a massive wafer can be done on multiple smaller wafers. The question is, are the compromises to get something working on a massive wafer killing performance to the point where just splitting the problem up efficiently would have been better. It's also important to think: Stuff isn't just going to happen to be the size to fit on this wafer, it's either huge in scale, or not. If it's not, you don't need this wafer, if it is, you probably need to partition your problem onto multiple copies of this wafer anyway. Take their comparison to a Nvidia GPU. It might be 57 times bigger, but I can rent 1000 Nvidia GPUs from AWS at the drop of a hat (more or less). So yes, maybe they've done some interesting stuff, but they need some decent benchmarks to show off before we can really distinguish between whether this is the Boeing 747 Dreamliner or a Spruce Goose.

streetcat1 7y ago
The problem is not if it bigger than nvidia (which is kind of a strange metric). It is when Amdhal law kicks in. https://en.wikipedia.org/wiki/Amdahl%27s_law https://en.wikipedia.org/wiki/Amdahl%27s_law
dgacmu 7y ago
You get much higher bandwidth (at lower power cost) between chiplets than if you used a multi-chip design or even an interposer. The drawback is that you suffer a really interesting thermal problem and have to figure out what to do with the wafer space that doesn't fit in your square -- probably creating lower-scale designs that you sell. The second drawback is that you can't really match your available computation to memory in the same way you can with a more conventional GPU. So you have to be able to split your model across chips and train model-parallel. The advantage is that model-parallel lets you throw a heck of a lot more computation at the problem and can help you scale better than using only data parallelism. Model-parallel training is typically harder than data-parallel because you need high bandwidth between the computation units. But that's exactly what Cerebras's design is intended to provide. You also have a yield management issue, where you have to build in the capability to route around dead chips, but that's not too nasty a technical detail. But if your "chip"-level yield (note that their chip is still a replicated array of subunits) is too low, it kills your overall yield. So they're going to be conservative with their manufacturing to keep yield high. It's not obviously broken, but it's certainly true we need benchmarks -- but not just benchmarks, time for people to come up with models that are optimized for training/inference on the cerebras platform, which will take even longer.
- nrp 7y ago
  Why even make the final giant chip rectangular? I get that the exposure reticles are rectangular, but since this is tiling a bunch of connected chiplets, why not use the full area of the wafer?
  - petra 7y ago
    You need to make sure you're io lines are imprinted on the edge of the wafer. It's much easier to do by making the io lines at the ends of each "chip" , vs a circle that cuts in the middle of the "chip".
pirocks 7y ago
Sorry for being pedantic , but there is no Boeing 747 Dreamliner, perhaps you meant 787?

streetcat1 7y ago

The problem is not if it bigger than nvidia (which is kind of a strange metric). It is when Amdhal law kicks in. https://en.wikipedia.org/wiki/Amdahl%27s_law https://en.wikipedia.org/wiki/Amdahl%27s_law

dgacmu 7y ago

You get much higher bandwidth (at lower power cost) between chiplets than if you used a multi-chip design or even an interposer. The drawback is that you suffer a really interesting thermal problem and have to figure out what to do with the wafer space that doesn't fit in your square -- probably creating lower-scale designs that you sell. The second drawback is that you can't really match your available computation to memory in the same way you can with a more conventional GPU. So you have to be able to split your model across chips and train model-parallel. The advantage is that model-parallel lets you throw a heck of a lot more computation at the problem and can help you scale better than using only data parallelism. Model-parallel training is typically harder than data-parallel because you need high bandwidth between the computation units. But that's exactly what Cerebras's design is intended to provide. You also have a yield management issue, where you have to build in the capability to route around dead chips, but that's not too nasty a technical detail. But if your "chip"-level yield (note that their chip is still a replicated array of subunits) is too low, it kills your overall yield. So they're going to be conservative with their manufacturing to keep yield high. It's not obviously broken, but it's certainly true we need benchmarks -- but not just benchmarks, time for people to come up with models that are optimized for training/inference on the cerebras platform, which will take even longer.

nrp 7y ago
Why even make the final giant chip rectangular? I get that the exposure reticles are rectangular, but since this is tiling a bunch of connected chiplets, why not use the full area of the wafer?
- petra 7y ago
  You need to make sure you're io lines are imprinted on the edge of the wafer. It's much easier to do by making the io lines at the ends of each "chip" , vs a circle that cuts in the middle of the "chip".

pirocks 7y ago

Sorry for being pedantic , but there is no Boeing 747 Dreamliner, perhaps you meant 787?