4 ms·

> 3) Constant multiple advantage over GPUs - maybe 5-10x compute advantage over off the shelf GPU, and that number isn't increasing with each generation. The a

> 3) Constant multiple advantage over GPUs - maybe 5-10x compute advantage over off the shelf GPU, and that number isn't increasing with each generation. The advantage of this hardware isn’t just the raw compute capacity. It’s the massive IO bandwidth. GPUs are fast, but they’re not designed for massive all-to-all communication across large networks. That’s where modules like this shine.

lostmsu 4y ago
Why do you need all-to-all? With deep networks you could just put different layers on different GPUs, then run the forward and backward passes asynchronously. They would only need to communicate in a chain-like manner.
- atty 4y ago
  Just as a quick example (there are lots of other examples), I currently am working on a model that does significant tensor-level parallelism. The input to the model is huge, so we replicate the weights for the encoder part of the model over 12 A100s, and then run an all-reduce over the outputs before then distributing that reduced tensor to another set of GPUs for the next set of operations. In this setup we need significant bandwidth not only between every device in a node but also across nodes (infiniband).
  - mirker 4y ago
    Isn’t the simpler example data parallel training? That’s a reduce and broadcast.
    - atty 4y ago
      You’re correct, I was going to add on to my answer that this is combined with DDP for a “3D” style parallelism that could specifically benefit from the TPU’s network topology, but by the time I got done fixing all the typos (and still missing a few) from writing it on my phone I completely forgot :)
  - Robotbeat 4y ago
    To be honest, I think regulators should have stepped in to stop the acquisition of Melannox (Infiniband) by NVidia. There are basically no non-NVidia options for Infiniband now. (Infiniband being about half an order of magnitude ahead of Ethernet at any one time, with lower overhead and lower cost per unit bandwidth. Often the basis for high bandwidth Ethernet cards as well. And ostensibly an open standard.)
    - atty 4y ago
      I agree that I’m uncomfortable with the consolidation of so much of the HPC stack under Nvidia (and would be 10x more uncomfortable if the ARM acquisition went through), but I don’t think there was really a monopoly there. I think the real “market” is networking, not just infiniband, and in that sense there is a ton of competition, just no competition that hits the performance characteristics that Nvidia/Melanox achieve.

lostmsu 4y ago

Why do you need all-to-all? With deep networks you could just put different layers on different GPUs, then run the forward and backward passes asynchronously. They would only need to communicate in a chain-like manner.

atty 4y ago
Just as a quick example (there are lots of other examples), I currently am working on a model that does significant tensor-level parallelism. The input to the model is huge, so we replicate the weights for the encoder part of the model over 12 A100s, and then run an all-reduce over the outputs before then distributing that reduced tensor to another set of GPUs for the next set of operations. In this setup we need significant bandwidth not only between every device in a node but also across nodes (infiniband).
- mirker 4y ago
  Isn’t the simpler example data parallel training? That’s a reduce and broadcast.
  - atty 4y ago
    You’re correct, I was going to add on to my answer that this is combined with DDP for a “3D” style parallelism that could specifically benefit from the TPU’s network topology, but by the time I got done fixing all the typos (and still missing a few) from writing it on my phone I completely forgot :)
- Robotbeat 4y ago
  To be honest, I think regulators should have stepped in to stop the acquisition of Melannox (Infiniband) by NVidia. There are basically no non-NVidia options for Infiniband now. (Infiniband being about half an order of magnitude ahead of Ethernet at any one time, with lower overhead and lower cost per unit bandwidth. Often the basis for high bandwidth Ethernet cards as well. And ostensibly an open standard.)
  - atty 4y ago
    I agree that I’m uncomfortable with the consolidation of so much of the HPC stack under Nvidia (and would be 10x more uncomfortable if the ARM acquisition went through), but I don’t think there was really a monopoly there. I think the real “market” is networking, not just infiniband, and in that sense there is a ton of competition, just no competition that hits the performance characteristics that Nvidia/Melanox achieve.